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ABSTRACT 

This book on evaluating social programs is an outcome 
of the Job Training Partnership Act (JTPA) Evaluation Design Project, 
which produced a set of 10 guides for the evaluation of state and 
local JTPA programs. This book distills ideas from these guides and 
applies them to a larger context. Part 1 presents a general approach 
to program evaluation and includes the following chapter: "Evaluating 
Programs" (Blalock) . part 2 presents complementary approaches to 
evaluating programs and includes the following chapters: "Evaluating 
Net Program Impact** (Johnson, stromsdorfer) ; "Evaluating Gross 
Program Outcomes" (Simpson); and "Evaluating Program Implementation" 
(Grembowski, Blalock) . Part 3 presents organizational and political 
issues in evaluating progrcjns and includes the following chapters: 
"User-Centered program Evaluation" (Feldman) and "Strategies for 
Supporting Comprehensive Evaluations" (Blalock) . Each chapter 
includes a reference list. Appendix A provides an overview of the 
JTPA, the case example used throughout the book. Appendix B addresses 
the capacity of program management information systems (HiSs) to 
support program evaluation. Appendix C lists the series of evaluation 
guides produced by the JTPA Evaluation Design Project. Appendix D 
provides bacxground information about the book's authors. (CML) 
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InMstmiofif^mnotiimtedmdogy, }w have little trouble dunk- 
ing aboM the comrUHstixm i^Tphisticated it^fimnation sterns 
u> the managemem ofprogramsn But we know tfmt ambers do 
mH replace gtwd fmrnagemem pn^ctices or the development of 
policies res^ponsive to the needs of the fm^e social programs are 
designed to help. The technology we need to thirtk inonp about in 
this nrs^pfcf is sodal science researdt, Althmtgh mamgers have 
always had responsibility for judging the adequacy cf their pro- 
grams^ dwy have ncf dwa^ lusd relitMe , practical U)ob mth i/i^dch 
to evaluate them. 

We now need evtduation tools rwt mly fi^ stm^ng the return 
m the public's investmem, Imt tools h^cA help us nuiru^e (mr 
programs better. We need detailed analyses of each cf the parts 
<mi processes mthin a program^s system. We need to see where 
we are most effective and where we can make improvements. 

When we Mnk of e^wttitm tedmt)logy as a management tool, 
we see it as stmething aqxMe cfht^ng us acctmpli^ tasks duit 
would be very difficult or impossible to achieve mthout that tool. 
It should be easier for the manager to make decisicms. The infor- 
ma^ Umt comesmttofan evabmion effort ^uM provide direc- 
titm in setting policies and establishing procedures. 

Bm we s^umUimdersumdt^ the use of ewlmuionmtty involve 
change. i4i»/cto^6ri^5m^ 

mth amducting evalmtitms^ we should al^ renmnber the risks 
take in run using wfua applied social res^trch can offer us in 
understanding our programs letter. As Qmgress and the Presi- 
dent develop plans to balance the federal budget, the pressure will 
be tmaUihm^ticproffams to fHove their worth. Iff^^techtK^ogy 
frfevaluaticmambeusedtohe^us, we ^^^umld setimisly amsider 
making ti^ best use of it we can. 

Uiwh Turner, CcMiunisssmer 

Wadiingttm Smc ^nfrioymrat Security Dejwm^nt 
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Chmges in tte organbatkm vid operation of {H^licly suf^rted scxdal 
programs in the l^K)s have imnroeed state and kx»l responsibility for 

fyr Mml repcmii^ purpo^, howevo*, im often proved iucnfficieitt to 
(^M^ the broader credibiliiy of ttese progmms or to gukfe cteeisicw abrat 
bow tD adjust {nogram pcdici^ or inqnove i^ograms Bt those levels. Al$o« 
ahtougb evaluatum r^earcfa \m bmi gei^iaUy accepted since the late 1970s 
as the least bumsed means of ertaMishing program accoiimdnlity^ most state 
and local service organizations have k^ked Ui^ fkKls and specialized exper- 
tise to develop die rngmng evah»tion ciqsability now neected. Tterefore, tte 
1^ evalimion aq>e(S^xB and increased resoim^e cm^^ 
tuive created a do\^ c&lemma. 

This dilemma {myvided tte context for a national evalu^on {»o^. 
the JTPA By^UiMion Demgn Prvject. butiAod, developed, dsA dtrocted by tt» 
Washington State Enqdcqmmit Security Dcpartn^m, tte jmiject reqxmcted 
tt) new state and local omsight c^lig^kms autiiorized by the Ic^ 'Haiinng 
Pbrtnership Act (HTA). Innikally, alUxHif^ ttm leg^atton mandated pro- 
grm evaluatim ai ihe s&Oe levd, it stdsstamii^ 
11^ conq)ared widi pievkws effiirts targtf^ 
»dpoorMdft|^awdtigte rK arict k Hisonttei»eofd«s 
ttmi^ incliHling im>gram ovenught. AltbcMi^ states c(^ use a sn^set-^kie 
fo* technkal mwtaasx^ imter «4ridi evalimikm oould be ratkmalixed, this 
money w» formally to tte )m>vision of fisral iiH^tives and t^duucal 
assifiai^ sai»:tioQs related to the level of state and focal cci^ 
formai^e lequimx^nts. 

Far mnoved from the afThioit days of iht 196Qs md \9J(h, focteral 
evaii»iion money fioi9i^ idioivdy finely to 

tkms, and still strcmgly affeoed by the legacy of state ami bod (^)emtence 
c»i federal research dfi^s ovtx th^ im{»essive period* the new mandate teft 
moM states struggling to define and the new cMigatbns they had so 
preci]»tiK»ly iid^rited. Washh^ttm Stete no ex(^i^. 

An initial c^iaBa^ was develop new autimMted inforamk^s sy »ans for 
wfetpiately nK^ntoriiDg de«^»nUized pn^rmns agamst b^ic fed^ re- 
q ui itt ne t it s. Inthe[m>oc«of<te^gnii^anewn Mn^em cminfi^^ 
(M£5)fi»'theia^d^Waidiington, the cmmiisdcsierc^ the Eopfoymeitt Securi- 
ty Dqxtrtntent requi^ted an issi» pvpm aitticiptfing inform^ needs for 
evaluatkm activities. A monograf^ cm the meaning ami value of program 
evaluatkm» and tttt kinds of data ami Informaticm system fe^ures it wmakibe 



Ukdy 10 iowlve, becani^ the bas^ of a sencnis agmcy effort to seanrh 
omsiife foods to sujqxHi ovmight ftctiviti^ beyond die simple mcmik)ring of 
im>grBra8 to ffetoroine ccmqrfiamre with fecferal reqpiirmimns. 

The fond-seaitdti (mKess nicrrased agoicy awareiM^ of tite coiisti;diits felt 
l^otbNirtatesiBitevdk)piii£anevahiati^ Commis- 
sbD for En^loynrart I^^icy e%pws9&i a similv concern timt ifame directly 
ra^)OiBib}e fi^-tte success of JTPA programs vmn ifwte<pialdy equipped to 
cvahnae their progiwtt. The Cmmifi^^ 

iropoad from the Emptoyn^ Secur^ IXya i tiiit a tt to dcvdky evahi^km tools 
!9)ec^k»Uy c»ki^ $0 the dii^^ 

The Comini^skm p^tteulaity adanwtt tlmt ttese tedmk»l assistance tools 
i^ve as pmciicai gukks fc»* jitc^qg die value of JTPA in ti» cc»ttext of a 
ifaunalicalfy mh»»l fed^ evahiMion {x^es^nce. The result was ^el(^ 
meitt of a {m>ject axx&pi by Wadungtoo ^ate, whidi vm subsequcmUy omStcd 
is6o a pit^x>sal for pc^lble joint funding. 

This cono^ inviHved tl:^ of two sets uf a»iq>lanexituy evaluation 
goi^— erne reiqx»isive i»inianly U) sla^ u»rs, os^ to locid. The guides wen 
tt> be c(»i^)etent sctentsfrally ami i^ful pragittttically. Their titration to 
ragoii^ JTPA pro^raim was expe<^ to intichK^ ol^ective inforniatim of 
direct and tsnely vahie to {m)gram ctocisicnmiakers at die leairt pos^ble cost* 
The craq^enmsttry nature of ttese tools and the oonamitiTCnt to competence 
s ugy st ed die jnes^cticm of an irnmSsc^naryffwipcf evaluation iksiffwrs 
Hiiow range of training and expose wmild im>vide d^ diffeient perspec- 
tives needed in stixfying |»nogFam mq>toimitation ani inq)^. 

The iia»ei^ in {mignmtism sugge^ed thtt Uiis ^un be setocted within 
Washingtcm^ if possible, to (tenxmstrate the g^ieral ability of state and local 
areas to inquire resMich ^vice withm or near ti^ own bonte^, from 
letKmces Mch as dmr own Aate university s^t)^ 

Viewing die ctesignm as a mearA panmrship was also consi^ent with die 
major feature <^ JIT A « naimly d^ pimsermm of ultin^ comoi widi publ ic- 
private partnradiq^s at ^ state ami local level 

To enhance die toUity of gui<tes, die Emphiynient Security Dqwiment 
feh It w» inqxiitam to add a to die seri^ tlMU suggo^ ways in which 
states and local areas coukl deveksp a fisc^» oganoatkmal, a^ 
to eng^ in cn^cmig evaluation activitks as an integral pm of infective im>- 
gmm managSDmit. Staff within die Emf^nwni Security Department who 
wereexp^ieDced in iumiseaieh» o]^nizatimalde^^Io|»n^, and evalutfion 
research, and fiumliar wiih the inmiendos of c»g^^ 
to tte r^nrdi partnenhqi. In diis way a dtengn team was form«i diat 
str^igdiraed bc^ dw qi«dity and utUity of the guictes and 
beneficial learning expolence for die team and tte agency. 
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Agjain consist with the JTPA partt^^p {mncipte, a piMtc-isrivate cm- 
^rtium cf poiemiaijwuters wi» setected and became part of Urn project pro- 
posal. Hie NittkHial Ccmimisskm fx Empkiynmtt Policy was rectmimmM 
as the JTPA Evalmaiim l^sign Prefect's natii^iid ^poi«^. 

The CcmuniiNium im a r«^)ected research reputtt»)n, and m panicipatkm 
laviaiti^ the imiject^s pcHemial usefulne^ to ittate md local areas« Further- 
mm^ die ConmusskHi viewed tte im^ed as a way to stiontlate intoiest in 
state and kmil evahmticm cm«»iaem wtdi the Act, ami to assist ^at^ and ser- 
vice delt\^ areas (SDAs) in developing ^ i»pabUity to |mxtuce infmnm- 
tkm thrt was sufiiciattly con^par^ite to nttke a contrilmtkm to mokmai train- 
policy. 

To extend die {xutMrship ktea, {nivate cmpome fiinds w^ soi^ to nvrtdi 
pot^ttial contrttHititm frvm the Conmiis^on and tte Enqrfoyn^tit Security 
DqpartmenL Since tl^ prk>ritie$ of ffiM*s Coiporate Suf^xnt f^iogran^ for 
1986 iiKrluded policy research, IBM was selected as die tey {Hivate funding 
partner. Its stature i» a nuiltin^oiml cwporatic^, and its interest in 
applying private sector qiudity omtrol mnd r^earch aiMl develqmient ideas 
to die public secunr ccmveyed spccia] credfl>Uity. Eadi of these tfiree fumters 
had son^thing distinctive to ccmtribute to the |Hoject. Hierefone^ eadi w^ 
reqiKSted to Insist widi a different o^mponem of an int^r^ed 

Submtttii^ an unsidictted propma) to a iiatkH»l funder md i^^lecting 
research ami fui^mg partnershqiis coo^tuted radical bdiavbr cm the part of 
a ^ate agency and posed certain risks. However, in making d^ r^n^te for 
ooUaboratlOT aod a divhnon of r^xnisibiltty anxmg funding partners explicit, 
potential fundcrs ki^w j^edseJy whm product diey be mb^^mng, and 
what their role vms to be in the lai^r ^htt, Uhiniately, the {m^x>sal was 
accq:^ , and all of tte mmdbers of tl»r ftnuiing and s^fHng [wrtnershii^ par- 
tkipated in tte im>ject as {Hx^iosed. In adcfition, impOTtaitt public-prtvate in- 
kind comributions were obtained from the Safmi Insurance Com(»mies of 
Anoerii^; tiic Wttwic^ Intemadonal HcMds, Ii^.; SI^, Inc«; a n^tioi^ soft- 
ware coiporttion; and die Setftte-King County Private IikIu^ CouikiL 

The Natioiml Ccmunk»on selected a hodorud adwary comnUttee fw the 
pn>|ect. The committee was cc^iqx»^ of ^ate and k3(^rqm»^^ 
six stat^, and supplenm^ Ysy ref^i^eirtatives of die majw n^onal fmblic 
interest grraps involved with ^i^^?yroent ai^ training aiid a mmdier of 
tionally known evahiatkm researcrhm. Thiscmuntttee|»ovkied initial u^j^ 
abtnit evaltMiiw h^ues of partiodar concern to d^ JTPA ccHnmunity across 
the coumr>' and revhn^mi mcomive drafb of the jmigect^s evaluation gukfes. 
Tl;^ Enqdoym^ Secttrity Department or]^b^ an in-hmtse ad\isary group 
to obtain ii^mt from Wadungton's and local traimf^ cnuKals mi 
gram dire^m. These mivisory grmps became an essential part of the pn> 
ce^ dirragh whbh die ser^ of gtiicfes was devek>ped. 
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In 1986, tl»: jHiojcct pn3chK»l two sets of 10 volun^ m:h, tm for slate 
ud cme fi^ local ii&cts. Tb^ volutin a>vered evaluation planning; process, 
gross outeofli^. and in^iact evalui^ns; a:M4)e]^flt aim^sts; quasi- 
expermienta} vs. experimratal research <fes»gns; and MIS issws to be i»ed 
in evaiuatij^ JTPA. The s^ of n^btrrm diiaritMittd witfmtf cost to all 
stale and kx^ JTPA directors throng pint iimiing fmm tlh^ Ccmimi»ion 
and the Enq^iymcait Security Departm^. and ore now avaifadsie on ntk^oficte 
in ti^ ERIC system. Hie volumes and tlH^r auttors are li^^ed in Af^pendix C. 

Wt^ the volim^ were conq^lcted* an miditicHml proposal was functed 
the U.S. Dq)artn»uofLdx»-8i«it)^IBM€^»pc^Btk»)to^^ 
re^k^ evaluatkHi work^dx^ based on tte evaluation guides for JTPA 
tttk>ners and U> cfevelop a new mtsonal e^uMion jourittl^ Evalitmkm Forwn, 
which focused m state asKl local JTPA evalui^n issi^ and »rtjvftkrs. In- 
ksml comributioi^ to these additional project levities ime tmk by regional 
office of the I>^mtmeQt of Labor, tlw Natk»»i Alliance of ^iness. the 
NatMHml Governors' Associaik)n, and the Naticml Associatbn of Cminties. 

This book is an ^tonpt to distill kleas from imoject materials produced by 
this unique umtetakmg that can be of gemral assistance in evaluating social 
prcqpams. i^iog JTPA as the case example. In this way the contributions of 
a diverse gxwp im\t been refocu^ to benefit a larger audience. 

Ann Bonar BLIock 
Projea Deveii^r 
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to 

Program Evaluation 



ThjH beginning section discusses the particular philosophy 
and approach to state and local program evaluation that serve 
as the framework for the book. The context and purpose of 
evaluation at these levels and the basic scientific principles 
and methods guiding evaluation activities are also expioiiKi. 

A user-oriented, comprehensive concept of evaluation is 
described. Its purproe is to provide complementary informa- 
tion about program implementation and outcomes for im- 
proving ongoing social programs. 

TT^ oi^anizalion of the book, and the use of a practical 
case example throughout, illustrate and apply this concept 
of evaluation. 
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Evaluating Programs 

Ann Bonar Blalock 
Washington Employment Sevurity Defhirtnwnt 

Decisionmakers , planners, project staff, and {program} partici- 
pants are increasingly skeptical common sense and conventional 
wisdim as sufficient bases upon which to design stnial pn}f{rams 
that will achieve their intended goals, 

Peter Rc«si and Howard Freeman 

Evuluafhm: A Systematic Appnnich 



Questions have always been raised about the value of tax -supported 
program strategies for rcsolving major problems in society. Resistance 
and inability to provide evidcnee of public value have, in the end. led to 
social unrest and sometimes active pmiest. ITiis is not likely to change. 
Throughout history, those receiving and spending public money have had 
to guarantee that a reasonable tradeoff would be honored between their 
rights and resfHrnsibilities. Discretion and contn>l have had to be balanced 
by an accounting of the public good produced, of Ht^ kinds and in the ways 
intemkd. Planned change has always required accountability. 

There is a new element, however, in the historical exchange between 
public privileges and obligations, h involves the fryrm accinintabilily 
now takes. The scientific t^hnology for developing evidence of value 
and popular support for using scientific approaches and methods have 
changed radically over the last three decades. Increased public educa- 
tion, breakthroughs in the automation of data processing and analysis, 
and growing interest in the iDle of applied social science research in 
weighing the risks and benefits of social initiatives have converged. 

Evaluators can now perform statistical analyses of large amounts of 
data rapidly and relatively inexpensively. Evaluation research has 
become an establ ished field within applied social research ( Paiton 1 986 ), 
and the r bli :*s sophistication about the contribution evaluation re- 
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search can make lo moa* objective judgments of socia! pn>gram!5 has 
grown significantly. Most elected olTicials, government professicmaJs 
and otheni responsible for social programs now accept science-based 
evaluation as an integral pan of public policymaking. 

Evaluaticm is less well-understood, however, as a pra^manv manage- 
ment uwl for making general imprDvemenls in programs and testing new 
ways to organize and deliver services for particular client groups. At the 
state and local level evaluation has not been fully utilized as an 
opportunity to infuse the policy and planning process with better infor- 
mation, but there are cogent reasons why this condition has prevailed, 
and why the situation is changing. 

While the value and legitimacy of evaluation was reinforced, the 
conservative social {X)iicies of the 1980s substantially changed the 
Un aUim of respimMhiUty for making judgments of the value of social 
programs. Control and responsibility were transferred downward* from 
federal lo state government. The premise for this change involved a mix 
of new and older political philosophies: new fiscal conservatism 
emerging in response to economic, ei^rgy, and envinmmental problems 
and postindustrial changes in American society; older conservatism 
favoring states* rights and oppositig **big government**; and the more 
idiosyncratic social welfare policies of the Reagan administration, 
designed primarily to shrink federal involvement in social programs 
altogether (Palmer 1986). However, this shift in the locus of accounta- 
bility occurred in the context of little prior slate training or experience in 
performing the tasks required by the new evaluative role. 

Although the federal evaluation effort throughout the 1970s was 
extensive, it did little to enhance state-level evaluation capability.^ 
Collecting tt^ data essential in carrying out federal evaluations required 
the cooperation of state and local program professionals, but these 
professionals were rarely brought into the effort as partners, and the 
information coming back to them from federal studies offered few direct 
informational benefits in adjusting policies or improving programs. 

Although states wenr required to develop automated rejwrting sys- 
tems responsive to fedei^l program goals, few incentives existed to 
incorporate an evaluation capability within their design, or to develop in- 
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house evaluation cxpcnisur. When pciiiica! changes nxjuia'd that states 
assume evaluation R!S|K)nstbil!4Xs, the infmsttucium to support these 
activities was too often missing. Nevertheless, rew pmvocaiive state 
and local oversight opportunities accompanied this unanticipated trans- 
fer of authority. Gradually recognizing these opportunities, stales and 
local areas have begun to take a new interest in evaluation as an 
information tool 

The Purpose of the Book 

This book is written for those at the state and kxral level who find 
themselves working within this new social welfare envin^nmeni: those 
who develop program policies, oversee their translation into ongoing 
programs, and seek to judge their vahie. 

Its purpose is to encourage them to viewcvaluationnot simply asanew 
responsibility they now find themselves required to assume, but as a 
pragmatic policy and management tool that can directly inform their 
(tecisicMis and improve tl^ir programs. It., purjwsc is also to provide 
those given the responsibility for judging the value of state and local 
programs with practical guidance about how to develop or enhance their 
ability to evaluate, and offer them assistance in planning and carrying out 
program evaluations. Although these purposes are oriented primarily to 
the information needs and capabilities of those most involved with social 
programs at these levels, it is hoped that the book will be of interest and 
use to state and local elected officials, the evaluation research commu- 
nity, public interest and client adviKacy groups, and ultimately the 
American taxpayer who makes social programs jx)ssible. 

The new public-private decisionmaking fwtnetiihips. and the pnv 
gram administrators and staff who are now more responsible for seeing 
that programs meet their intent, have always wanted to be effective 
decssicHimakers. Often tlnry have not had sufficient in-depth information 
about their programs to feel comfortable about the decisions they must 
make. In an em that affords states and IcKral areas vastly increased 
opportunities to think more imaginatively about defining accountability, 
evaluation is viewed in this book as a chance to obtain a better under- 
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stamiing of the much more complex and intriguing interplay of forces 
within programs, ami between them and their envirDnments. This 
understanding is necessary to the souikI decisions practitioners want to 
make. Approaching cvaluaticm as a new opportunity, states and local 
areas can use their increased autonomy to coasider what information is 
essential to them in enhancing programs, particularly at the level wherc 
critical relationships between client and service provider occur. 

Using Employment and Training Programs as a Case Example 

Given the book*s purpose, and its focus on the information r^eds of 
state and local program professionals, selecting an ongoing program as 
the key illustrative case throughout the bcK>k was considered a m^.^ul 
device for giving abstract concepts a ncher and more practical meaning. 
Therefore, in each chapter the general ideas about evaluation, which arc 
relevant to a range of social pn^rams, aie applied to a single ca^^ 
example. 

The basic program for adults authorized in 1982 by tte federal Job 
Training Partnership Act (JTPA) was selmed as the illustration. JTPA 
exemplifies the contemporary trend in social programs, and its intrica- 
cies are well-known to the book's authon;. T^^ stimulus for writing the 
book was their <kvelopment of a of complementary evaluation guides 
for use by state and local JTPA council leiders and program practitio- 
ners.^ Preparing the guides required a thorough knowledge of the 
legislative intent of JTPA and the actual operation of JTPA programs. 
This familiarity has permitted the authors to apply the concepts in the 
book to realistic program situations, dmwing from actual employment 
and training program evaluations. 

Frustrated by criticism of previous social program efforts to resolve 
unemployment and underemployment among the disadvantaged, and 
needing a bipartisan agreement on a new strategy, tlw? architects of JTPA 
designed a program that made only incremental changes in the goals, 
services, ai>d target groups of traditicmal employment and training 
programs, but substantially changed the philosophy and fiarr^woiit 
within which such progranis were organized and implemented. 
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Tte propose clmnge agent in JTPA was the ran^ of basic services 
common to previous appro^tes, with the major omission of public 
service employn^t ami the restriction of training stipends to thc^ most 
in need. The eligible client population remained t!^ unemployed, the 
^^ommucally disidvantaged, tli^ officially poor, and ttose with signifi- 
cant barriers to employment. The philosophy underlying JTPA was 
unique am! antici{»t^ future tremis. This new initiative involved 
reduced funding and tte d^ntralizaiicm of power and responsibility to 
imbltc-pnvate parmershi}:^ at tl:^ state and local level. 

For the first time in dK history of national public employment and 
training {Hpgrams, and consistent with tl^ strengthened role of the 
imvate sector, specific methods of quality ^ntrol, i.e., federal perform- 
ance standards that were to be met or exceeded by \oq^ service delivery 
areas (SDAs), were made an integral pan of the legislation* State fiscal 
rewards and state sanctions in the form of mandatory technical assistance 
were incorporated into the legislation as reinforcements for local confor- 
mance with standards. 

JTPA also required increased coordination with related programs, and 
encouraged new modes of organizing the provision of services, includ- 
ing performance-based subconu^cting. Most relevant to this book, 
slates and local areas were expected to move l^yond monitoring pro- 
gram outccm^ against a limits set of performance measures. Tl^ 
legislation mandated that states ex'aluate their programs as welL Seem- 
ingly novel at (Ik time, many of JTPA^s distinctive features became 
models for future publicly funded social programs. TTie reader will find 
it useful, in understanding the application of the book's central ideas to 
JTPA, to refer to the more detailed description of this program in the 
appendix. 

The Framework of the Book 

Tte concept of evaluation that provides the framework for this book 
is not an orthodox one. Despite the prolific literature that now charac- 
terizes the special field of evaluation research, most b(x>ks and articles 
cm program evaluaticm have been written primarily for researchers or 
oriented to those (tealing with program issues at the national level. This 
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literature has only rarely ^Idiessed state and local evaluation needs and 
interests, or been directed to those making decisions about soci^ 
programs at levels clos^ to tl^ client. 

Ccmsistent with its purpose, this book describes a somewhat novel 
cmK:ept of evaluation that is specifically respcmstve to the information 
needs of state ami local program decisionmakers, Hiis perspective on 
evaluation recognise the expanded q)portunities inherent in their new 
oversight respmsibiiities, and resf^ts the singular constraints and 
supports for evaluation at the state and local level. It views program 
evaluation iKilisticaily as both an art and a science. 

Studying programs in real-life environments does not take pl^ in a 
social vacuum* It is impossible to isolate re^arch activities from other 
important influences. oiganizationaU political, and t^:hnical lurtivi- 
ties involved in producing information to inform decisions are interde- 
pendent. Giving attention to the scientific prirK:iples and metlKxis 
involved in studying prc^rams is critical. Restricting tte book to 
research issues would ignore the larger enviroimient of evaluati<Ht that 
resists or supports the application of research. Hierefore, the book seeks 
to integrate the technical aspects of evaluation with its organizational and 
political realities. 

The content of the book focuses on the state and local users of 
evaluation informaticm. the conditions under which they must make 
program decisions, and the kinds of elisions they are expected to make 
at key points in a program*s planning and fimding cycle. The assimiption 
is that the timeliness and relevance of evaluation informatic^i, as well as 
its accuracy and reliability, predict its level of use. Hiis does not presume 
that the mere presence of these qualities is sufficient to cbtvc a predcmii- 
nant role for evaluation in ^ping policies or directing the fate of 
programs. A strong visible role for information production of this kind 
is mitigated by many other compelling agendas. 

Nevertheless, there are key points in the policy and planning cycle 
where personal, bureaucratic* and poHtical influences ccmpete with less 
sub^tive sources of information to produce decisions through negotia- 
tion, exchange. aiKi compromise. The point in this decisionmaking 
process at which the worth of a program strategy must be judged is 
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paiticulurly vulnerable to the convergence of a variety of preswircs. but 
it is precisely at thi^ juncturc where competent program evaluation can 
have its day in court. 

its case will not be heard* however* if evaluation information is not 
directly relevant to ti^ content and ccmiext of the decisions being made, 
or if it reveals too little sensitivil/ to decisionmakers* infcHHiation needs 
ami technical soi>histicati(m. On the cHher hand^. in tii^ al^nce of 
empirically based usable information at diis decision point, prc^ram 
imustments will inevitably be based on more biased, ami potentially 
misguided, considerations. To the extent that objective information can 
serve as one important infiacnce among m^y other factors, slate and 
local policies and the prognuns they gei^rate will come closer to 
accurately defining and eHectively resolving social problems. 

In this context, the vimprehensive concept of evaluaticm proposed in 
this book has a number of dimensions. Each is important, but the strength 
of the book is in its attempt to integrate these elements into an overall 
evalmtUm strategy. Tl^sc dimensions can be expressed as interdepend- 
ent s^uential propositions. 

1. Potential users of evaluaticms need to be identified arni educated 
about the practical benefits of becoming informed consumers of 
evaluation research and using it as a decisionmaking t(H)l. 

2. Stale and local program councils ai^ agencies t^ed to develop an 
adequate organizatitmat capability to evaluate their programs — to 
initiate, plan, and implen^nt evaluations in-house or with the 
assistance of outside research consultants. 

3. Evaluations should be planned with the primary goal of improving 
programs. 

4. Evaluations will be of minimal utility in improving programs if 
evaluation sponsors fail to seek answers to questions that are 
responsive to the interests afid concerns of key decisionmakers. 

5. Relevant insights from previous program evaluations need to be 
consiikred in planning evaluations if the information from resemrh 
studies is to contribute to program improvement and to the accumu'- 
lation of knowledge about scnial programs from which division- 
makers can draw in the future. 
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6. The mast useful knowledge ba.se for decisicmmaking is one that 
integrates information trained by using amiparahle evalnatiim 
approaches and methiniologics across different programr 

7. To be of maximum use in improving programs and ccMitributing to 
general knowledge production, compambie evaluations should yield 
complementary information about (a) program implementation, 
and (b) the relatitmship hetywen seryices and outcomes. 

8. This complementary information will be more accurate and usable 
if stiKiies of implementation and impact are based on the use of 
common definitions and measures (jf the major variables, involve 
the same historical cohort of clients, reflect the same program 
context and environments and represent the same time period. 

9. If this complementary infcmnaiion is to be fully used, it must be 
presented to decisionmakers in user-oriented, nontechnical form at 
optimal points in the program planning cycle. 

10. Comprehensive evaluations providing complementary information 
to decisionmakers make the best use of new state and Imal oversight 
opportunities. 

Hie Meaning Evaluation 

in promoting thiv concept of evaluation, it is imixirtant to defme what 
we mean by program evaluaticHi. Like any complex system, the organic 
zatiraLS involve in implementing social programs are expected to 
perform certain essentia! functicms that assure that programs are contin- 
ued, can be fine-tuned to increase their effectiveness^ and can be adapted 
to changing client needs and circumstances. 

In exchange for funds, tl^se oi^animions are typically required to 
review program plans, monitor tl^ way those plans arc carried out, and 
evalmte the extent to which programs m»t their overall intent. TlK^se 
obligaticms. ti^ to fiscal support, are usually authorized by tegislation or 
the formal policies of governing bodies. 11^ contemporary form these 
expectaticms take is drawn from American industrial concepts. AlthiMigh 
it is infinitely more difficult to define "production process/' ^'consumer 
products,'' and functions such as "quality control'* and "research and 
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development'* when dealing wiih human -vervices. ihe same primriples 
that have characterized the industrial process have been applied to social 
programs. 

Certain consumers are selected as the key recipienis of the goods 
produced, particular steps are taken in producing these goods, a certain 
level of consumption is maintained, and tlx: consumer is affected by the 
product in some identifiable way. The characteristics of the process and 
inDducts and the costs and benefits of the elTort are observed and 
leconted. Marketing the product may be rw^ssary, and research and 
development are accepted as a basis for change in the consumers 
targeted, the nature of the proorss, x)^ product itself, the way it is 
introduced to the consumer, and the effect of the product on the recipient. 
Historically, ttese e>epectations have always attached to public expendi- 
tuies in cm form or another. We have simply made them more explicit 
and measurable and adapted them to the human service sector. 

It is natural, then, that social programs meet such expectations and that 
certain oversight functions be established to assure quality and accounta- 
bility. As the methods for carrying out these oversight functions have 
become more precise, automated, arKl reliant on scientific method, the 
distinctions among them have grown clearer and more ccmsistent across 
different areas of social legislation. In this decade, review, monitoring, 
and evaluation are gcneraliy understood to be distinct hut related 
accountability fimctions that have dilTerent purposes and require differ- 
ent expertise. Collectively, these assessments seek to preserve the 
distinctive features of a program. 

The typical differences among these functions are suggested in 
chart LI. Based on these distinctions, evaluation is defir^ as the 
systematic collection, analysis, and interpretation of information to 
answer questions about the efficiency and effecdveness of program 
implementation and impact, using the prim iples and methods of social 
science research (Rossi aini Freeman 1985). 

Wl^n we study a program's efficiency, we want information on how 
well resources are uj^d to achieve program goals; i.e.. are we achieving 
maximum social benefits for a given outlay of resources? When we 
investigate program effectiveness, we want to know the tradeoff between 
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nonmonetary costs mi hencfils as well as between monetary cosis and 
bcmrfits. Evaluation foeascs lun only on what is happening in a pfx)gram. 
but also why and how. Vk emphasis is on studying relationships among 
difTerem aspects of a program (Weiss 1972), 

In gathering infonnation on compliance for reiwrting purposes, 
mcmitohng provides information that h useful and M^metimes essential 
to an evaluation effort. However, it is not the purjK>sc of monitoring to 
collect the full range of data on the way ptxigrams ait? organized and 
carried out or on iht* relationship between their interventions and etTci^s. 
which is requireu to answer critical relationship questicms. 

An interesting exatnple of the differences between monitoring and 
evaluation is the mjMidaiing of oHicial performance standards lu monitor 
the outcomes of pmgmms. The use of standards is an efl'ort to formalij:e 
the provision of evidence that a certain level of ^wal achievement is being 
attained. The DepartnK?nt of Labor develop'.nl a set of standards^ 
measures for each of the standards, and a moniiori.ng pnxress to compare 
JTPA imiployment. inct>me, and welfare reduction outcomes and pro- 
gram costs against quantitative goals. Compliance y<i\h these standards 
is delemiined using a narrow set of easily accessible, lov^ cost, quantita- 
tive measures, such as the number of clients iilaced in jobs and *heir 
hourly wages. 

It would seem thai using performance standards is a form of program 
evaluation, or at least illustrates a gray area between mimitoring and 
evaluation. However, the perfonnance standards strategy is Hmiled to a 
set of proxies for a much broader array of evidence of program efficiency 
and effectiveness. Although wc may be making an important as.«^ssmem 
of apiDgnim'seffectsby monitoring selected outcomes against carefully 
selected but incomplete standards* we may also obtain a skewed view of 
a program *s value. 

Compliance with standards does not tell us anything about relation- 
ships among a more extensive set of potential outcomes, between 
outcomes and the organization of the program, or between outcomes and 
the way services are delivered to the client Most significantly, the level 
of CHjmpliance does not inform us abt>ut the program *s role W.v c/ iv v these 
outcomes: i.e., was i-ompliance ivsponsible for the outcomes, or were 
these outcomes due to chance or other influences? 
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Monitoring information is not intended to help us look in ckrpth at what 
is going on and explain ii (Blalock J 982). Unfortunately, however 
monitoring requirements tend lo pieoccupy state and kKra) adminiNtra- 
tors with a rcsiiictcd view of outcomes, distriHrling them from consider- 
ing the magnttucte^ quality, and durability of progi^m effects, and 
dampening tlurir natural curiosity -J^ul the "^black bo\^ that holds the 
secrets of how and why certain outcomes may have iKcurred. Conse- 
quently, decisions about adjusting pn^grams an: often made without 
sufncsent information. It is helpful, then, to think atxiut evaluaiitnt as 
distinct fiDm monitoring rcgarding the questions each is expected to 
ansv/cr (see chart 1.2). 

The Practical Benefitis of State and Lwral Evaluation 

Piictitioncrs have little difilculty appreciating the need for planning. 
man;^ing, and monitoring, or understanding the prerequisites and bene- 
fits i/:hese activities. But th ^y sometimes question what social science 
research can do for them thai collective experience and conventional 
wisdom cannot. Classic stereotypes about evaluation persist: it is too 
costly, risky, esoteric, inconclusive, too easily underutilized or misused. 

Evaluation J(h\s cost. However, the largest program outlay, other than 
personnel and services, has characteristically been the start-up costs for 
designing and placing in operation automated numi^ement information 
systems (MiSs) for requir**'! monitoring purposes. As the reliability of 
the information in these systems hos improved, and their technical 
sophistication for recording, storing, extmcting, and analyzing data has 
grown, interrelated liSs now afford a basic information bank for 
evaluation, which can easily and inexpensively supplemented with 
additional data, h is ir^lTicienl, in fact, not to use these systems as an 
evaluation opportunity. By making optimal use of the monitoring data 
already being routinely gathered, in terms of data analysis opportunities, 
evaluation can expand lesources for improving state and lcK:aI programs. 

It is true that cvaluatiims nay sometimes produce results that poli- 
cymakers and administrattM-s A^ould ratl^rnol know. Some evaluations 
may lead to the conclusion that :t program has made little or no difference 
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for some of the jKoplc exposed lo it. However, such a^sults are useful in 
modifying unrcalislic assumptions about the effeti a program's miplem- 
entaiion and service strategies should have, about the gmups for whom 
such strategies are lo v^ork best, or abi>ut the kinds of changes needed to 
make programs more efiective. Disappointing results can also alert 
practitioners to influences that should have been studied but were not, or 
to developing belter definitions and measures of the factors that were 
studied. It may be problems such as these that explain nonsignificant or 
negative rcsults rather than the characteristics of the pmgram. Aware- 
ness of such issues can lead lo a belter understanding of pn)grams and to 
morc inibrmed changes in the allocation and targeting of program 
resouavs. 

It is well-known thai evaluation findings and conclusions drawn from 
them are misused more often than we would prefer, but policy and 
management decisions clearly suffer more fwm an absem e of scicniil ic 
efforts to obtain objective information. More often than we think, 
evaluation offers new chances to demonstrate sutvessful pn>gram as- 
pects raa*ly rcvealcd by monitoring infonnaiion. Important outcomes 
that do not lend themselves to numerical measurement and are critical to 
a program^ success are frequently missed in the monitoring prcxress. 

On the other hand* evaluation shc^uld not be ovcniold. Science cannot 
establish "the imth" beyond a doubt. Scientific method only brings us 
closer to fhe truth than more subjective and undisciplined methods. 
Nevertheless, myriad, rich, warm anecdotal reports no longer convince 
taxpayers, program managers, or legislators that a program is worth its 
cost, or even that it works. Accurate empirical evidence of the value of 
both ends and means is the contemporary measure of the worth of swial 
initiatives. 

Therefore, the challenge for states and local program areas at the 
beginning of a new decade is to develop the deeper and more expansive 
knowledge they must have in order to make important short- and long- 
term program decisions: how to determin:f needs appropriately, use 
rest>urees wisely, target services so that those most in need or likely to 
benefit will have access to them, plan and manage programs effectively* 
coordinate with related programs in a meaningful way, and test new ideas 
as a basts for innovation. 
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An i?iiprcssivc bcncfl! of evaluufion in meeting this challenge is the 
reduction of bias in the infomiaiion made available Ibr decisionmaking. 
In tl^ simplest temis, a scienlific appnwich is a piide map for ohfahmif; 
the leashhiused mformutkm available about what is really happening in 
programs- Bias-frce information must be an essentia! pan of the range of 
influences to which decisionmakers exjwsc themselves in making judg- 
ments of pmgram value and recommending change- 
Administrators often take the position that evaluation has httle impact 
on the decisionmaking process* suggesting it is used mainly to supfHiri 
foregone decisions. This is often inie, but this conclusion neglects the 
role evaluation plays in nanowm\i the rantje of t hon es available to 
decisionmakers to those likely to he the best ones. The development of 
performance standards in employment and training pmgrams is again a 
good example. The measures selected to "stand for" employment, 
earnings, and welfare-reduction outcomes were based on a substantial 
number of previously undertaken national evaluations that tested the 
predictive ability of optional measua^s of these traditional program 
efTecls (Bamow 1987). Piose with greatest predictive capability were 
suggested as a basis for federal decisions on performance standards 
indices- Administrative choices were thus directed to the most accurate, 
Ieasl-bia.sed measures. The ultimate decision about siimdards favored 
these measures, benefiting fn^m the framework for decisionmaking 
provided by iasights from past e\aIuations.^ 

Changes made in programs arc inevitable, \m not a? ways desinthlc- 
Wilh increased stale and Icxral control, and the new flexibility to shape the 
purpose and content of change, administrators have the power to facili- 
tate the kinds of change that will dircctly benefit their pn>grams- 
Thercfore. it is critical that state and local decisionmakers have a pixil of 
unbiased information to balance other pressures for change. 

Reducing Bias 

Information about social pnigrams is hardly lacking, lire issues are 
the reliability and validity of that information. The practical benefits of 
program evaluation are tied to the need for (^Jei tive information for 
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decisianmaking. Imponani decisions about the future of programs are 
made on tlu? tmi^ of diverse information sources that vary in their 
approKimation of the truth about a program's impact and the effretive- 
TOTss of its implementation. Excisions are clearly more useful to society 
if they are informed by accurate information. Well-designed and 
competently conducted program evaluations that are res^/t *sivc to slate 
and local concerns ate the mc^t trustworthy source of infomiation for 
making choices among alternative courses of action. An impoitant 
purpose of evaluations is, therefore, to reduce informarhm bias. 

Biases can be viewed simply as factors other than those studied in an 
evaluation that may. or probably do. affect the answers we obtain to 
evaluation questions. The degree lo which such answers may be biased 
is a function of the extent to which an evaluation plan can control for 
extraneous influences that may explain ation results as well as or 
better than those being studied. That i ally we want to eliminate 
competing explanations of our findings, so that we are assured that what 
we have learned applies sj^ifically to the relationships we have selected 
for evaluation attention (Smith 1 98 1 ). 

Based on the principles and methods that characterize scientific 
method* the logical chronology of steps in the research process is 
designed to control for the effects of bias to the extent possible. This 
chronok>gy describes mettiods that are within the control of the evaluator 
and methods that are fallback strategies to compen.saie for potential bias 
outside the evaluator*s control. However, not even this systematic 
approach can eliminate all biases, since even in the most competent 
research many influences cannot be anticipated or measured. Knowl- 
edge about their potential effects may be lacking. It may be very difficult 
to quantify such influences for purposes of analysis. TTie cost of access- 
ing such information may be prdiibitivc. Organizational or political 
constraints may exist. Nevertheless, conforming as much as possible to 
the classic steps in the research process brings us closer to the truth than 
any other method (Blalock 1982). 

An overview of these steps is provided in chart K3. with tte caveat 
that, in practice* the questions to be answered in this progression are 
highly interde{K:ndem and must often be dealt with simultai^^sly. To 
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provi(te the reader wiih a clearer idea of ihe kinds of biases that may 
emerge at each point in the rescairh process, a hypothetical workAvelfarc 
demcmstnition project is used as an illustration in chart ! .4, 

The Range of Evaluation Options 

As we try to confomi to the research process, we should appreciate the 
alternatives available for evaluating state and local programs. A variety 
of distinctions can be made among these research options, but we will 
concentrate on two: (1) the issues given primary attention, and (2) the 
characteristics of the research Jesi^ins used to study these issues. 

Issues of Interest in Evaluating Social Programs: 
Studying Program Implementation vs. Program Outcomes 

Most major national evaluations, such as studies of programs under 
the Comprehensive Employment and Training Act (CETA) and the more 
recent series of evaluations of state work/welfarc programs, have fo- 
cused singularly on a program's outcomes (Mirengoff 1982; Gucron 
I986)» Recognition of the significance of studying program implemen- 
tation has evolved slowly, even though practitioners have always had an 
intuitive awarcness if its importance. To select among various evalu- 
aticai posstbilities appropriately, in a way that is responsive to different 
information purposes and evaluation circumstam^s, we need to under- 
siand the difference between outvome evahiations and studies of pro- 
gram implementation, or prtwess evaluations. 

The major impetus for the development of social programs lies in the 
political arena as much as in the realities of life in societies, Tte 
definition of certain social phenomena as "problematic'" is shaped 
strongly by the parameters of public det^tc, media attention, and the 
political process. Having been assigiKd problem status, pressures build 
to develop public policies to ameliorate or resolve these conditions. 
Some policies are subsequently tmnslatcd into action programs through 
legislation. Ttese programs are essentially strategies for change — 
change in individuals, oi^anizaiions, institutions and their environ- 
ments. Their purpose and content are progressively reinterpreted by a 
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in tl^lvabmUaii Proem Osestiom AsswotiI 



Coi^ptoalizattim and MMStromem 
Rririew of Pttt Evalintlons 

Identihcafton of msiflhts from past researcti relevant to the siibject ot 
interest. 

Identification of the assumptions underiymg ttifs research. 

identification of the important fssues to study. 

idefrtific^ion of the major mtabksio investigate. 

Identfficatmn of the most important refatmshfps on which to focus. 

Formulation of a manageat)}e set of top pnoniy res^rcn QoestHm^ to Oe 
answered 

Id^if fcatlon of the vanabies to be co^^troiM in ans\)^nng the research 
questions. 

Specification of the assumptions made about what is most essential to study 
iWu«tf»mftflt 

I>elinitlon of the vam^. or influences to !>e studied. 
Deveiopmen! of ^nctees for each of the vanables 

Detlnitton and devetopment of indices of the variables to be contrc^. 



What reasonably reitabfe and vaiid mformation does |»st research provide 
tfiat can help direct this evaluation? 

What assumptions did the evaluatofs make t^t can provide an effective 
ratiof^le for selecting questions to be answered m this evaluation? 

Wfiat prt^ram issues are most significant to study? 

What influences teve the greatest impact in shaping these issues? 

What relatk)nships among the major influences should be the subject of the 
evali^iCKi? 

What are the most important questions^ to be answered about these i^ues« 
influences, and rei^onships? 

What inf li^nces — ottmr tim f/rose fm stiMfy — may explain tt^ 
answers ^idwJ by the evatuati(Ki. and therefore must be given attention? 

How can the f(K:us of the evaluation be rationalired'? 

How will the major influem^es be defin«l in order to study them? 

Consistent with the definition of th^ \^raibt^. what kinds of masures 
(data elen^ts) wiJI be used to n^sentmt) influence in the evjyuation? 
How witi influents other tlimi what is bmng sWidmi be defined mi 
measured, m onler to control for their fwt^tial effects? 



Tte Ev^Btkm Cimmology 



St^ to tte&vMfos Promt bMiimAasvi^ 



Dev^oprront of a research approach, and a set of sampling, data collection 
and data analysis methods for answering the re^rch questions, given the 
data avsitatite. 

» Sanripltng 

Oeveiopmeni of a sampJing strategy for seating the subjects of the 
evaJuatkwT the sample of clients, and a comparison gro*ip (if 



• OataColl^ion 
Dcvelopnient of a strateoy for acquirmg the information needed 



What kind of research gunfe. or technfcal mdp, will be used to d<?ect the 
evaluation, so that tt will yield the least tmed mformalton? 



How should the mdivtduais, groups or organ i^ltons to be studied be 
selected, m order to be able to generalize the results of the evaluation? 

Hw clo^fy does the comparison group iif any) resemble the client group? 

What is the most cost-effective way to coifed a given quatity ?nd quantity 
of data? 

How should data on the nieasures selected be garnered, in order to obtain 
the most reliable information possible? 



» Data Analysis 
The development of a strategy tor analyzing the data collected, 
appropriate to the research design. 



What statistical or other n^ethods are most appropriate for analysing 11^ in- 
formation obtained, m order to produce the most valid results possible? 



Tim Bmimnw OmmAitn 



M|» to tto EfftliiMlcm Proms 



US8 0fR«raKs 



fommartoUon of Findings 

The presentation of the most important evaluation resutts. given the 
f^eardi qi^ions. 

MtrpretatimofFinifiitp 

The drawing of condusmns from the findings, qmn the Qi^lifications to 
these resufts. 



Mvite ebeot tmprovint tfie Program Evahiatofl 

Tf^ dewfopment of recommendations* about improving program policy and 
operation. 



What was learned about the issues, influences, and relationships of interest? 
Whac are the answers to the re^rch questicris? 



E. 

5' 



What conclu^ons can be 6rmn from the findings, considering the 
following • 

• What pot^tai sources of bias were not measure, or wen? outside the 
evaJuator's control that may limit evaluation conclusions? 

• How do the r^ure of the r^earch questions and the characteristics of 
th(^ studied affect tl^ usefulness of the conclusions? 

• To vrfiat extent wJnclusions be generalized beyond tfte subjects 
of this evaluation, and the context and environment of ttte program 
evaluated? 

Bas^ on the findings and conclusions, and considering the biases left 
unmeasured or uncontr(riied. as we!! as the extent to whk:h results be 
generalized, wtet recomme .dations can be made from the e^^!uatfo^ ttiat 
could improve program poiicies and operation*? 
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ftyteiUai Sosrm ol Bte tn Ewtafim i ftew MMten Rsftrat Proved 



Enmpta of PottsUal of Bias 



Review of f^st 



Research I^»gn 



If past e^uetlor^ of v»)fk/«^fta^ {m)grams are underutilized, or tf the analysis of previous M^rk^weHare studies does not give 
enough coi^uter^on to ^ nmrthodology used or the qu^itications that nru^ t» macte to the findirrgs, \^ evaiuation qi^stions that 
drive the evatuation of the ttemon^ratit^i niay t>e based on mtequate or fautty assunqrt^ and (Elusions; ti^refore. the 
answers nuay tecK utility or be mmlemJfl^. For example, %om (A the earfy e^Huatmns ot "worWare" progran« have product highly 
questJonaWe resutts. i^ier^ recent ejq)eriTrentaI studiw ot wrk/wtfare programs have yieWed much more tru^twoithy ir^ights. 



H the im{mt^, tess rriewtt. or mot^ vanabt^ ar^ r^dN^tships are se^ed for study, the usefulness ^ cost-eff^tiveness 
of evatuation activities may not justify the etf(^. For exampte. the better studies of w(^k/«^ffare programs sug^t that the marr^f and 
family characteristics of the unit to bertefited t^ a pr^ram. and servit^ ^ntGr^^nH>ns in^hmig h^ith services, day care, and 
suppoitive counsel^, are tey ^r»bi^ to nmsure in workAit^arc prc^rams. K^^vever, many stud^s have persisted in tooking only 
at the standard demographic variables and the effects of standard ^pliym^t and training services, protmbly musing the more 
important expianations for outcomes. 

If the essential service interventions, outDun». and'Or imptementation features that characlerire the welfare reform program — 
and are the influence of greatest intere^ m evali^ng it — ai^ r^t dehr^ preciseiy and accurately, the resluts witi tack valiifity and 
k)seth^rv3h^ in improve the r^ program. Fi^ran^le. in ^ud^tt^^^ of the nibc ami secKJ^^ of services p 
given cl^nt, ^ rs importam to dev^ a pmitse. ea^ umterstandab^ ctefmitkHi of what mi\ servr^ in the mhc and how sequence rs 
to be (fefined. For exampte, ttw ^rvk:e inlerventicKt 'd^(^m trainmg" has many poter^ m^ir^. Precisely v^t acttvihes it 
is to invrtve in the reform jm)gram mu^ be esliAl«I»d, and the n^ii^ of "seqiwic^ ^rvm" must be specified, if the effects of 
receiving th^ and (^rservm in partk:ul^^i^m;es« are to accuracy ctetenm^, 

Of^ set of tmes that are difftcua to control for, but mi^ be d^it with in d^'gning an evaluation, are s^ron tme^ These can be 
biases due to d^nts* s^-se}^tk>n into a program, or t^es dt^ to the ^ prDgram staff c^ry out intate. appra^, service 
iHannir^. serv^ ai^nn^nl amf nefena} r^>oi^fli^. At each of tte latter dec^n points, bi^es due to cer^ clien! and staff 
char^ristks, and i^rtetn service itelivery pr^t^ can interfere with obtaining valid remits. For exampie« among ^rglble only 
a subsd apply for a program. Wtfli ^ refonn, tf% applicant group may over^represent indmdt^ of a certain age. with cer^ kinds 
of motivation and prior sWIte, TJw ev^tor must «>nwhow compensate for self-scl^aton biases H ttw information itesrred from the 
evali^tion is to aM>ly to the (mtire grtmp of ^igtbtes. 



Potentel Sourm ef Bits in EvtbttOiqi i Itew WvlfarB HOtmn ^{ttl 



Evatoitiof) Proem 



((kmtmed) 



Sanding 



Datt Ceffection 



Data Analysis 




Esmmptts el fH)tefitiii Samt^ fit 8ii» 

Staff trequfifltly ass^ services on the basis of their 01^ sense of who n^ds and wil benef rt from particular options. irrestMJctive of 
fonnal policies and procedures — or formal ^\icm themseiv^ may require nonrandom assignment proc^ures In evaluating the 
reform, the evaluator must (tetermine what kinds of service ^signment bia^ are lik^y to to mtrodiK>^. so con^nsatory strat^ies 
used to redice these t^^. 

If the sampte of individuate sdectwl to be studi«l in evaluating the reform fs not representative of ttme about whom mtom^tion is 
desired, ^ results of H% evalu^ion v^U lack generali^^lity: the results wiQ only apply to a subset of those ^r^ipating in the 
program, and it may even l» difftcutt to identify precisely tlw kinds of imftviduals to whom the information does apply. For exampte, if 
we want to know the postprogram outcomes of all participants, tnrt the sampte we draw o^r^tects parttctpants with particular char- 
acteristics, m r,annot \tm concliHte that the outccmw hold for all partk^ij^nts — If^ may only hoW true for the most disadvantaged 
or the most skilled program ai^lii^nts. 

If me evaliartor wants to comjMre mdivWimls exj^r^ir^j me reform wSh ^milar indh^duals who are expen^cing ttw older program. 
amJ the comparson group is seterted in such a n^iwier that ft (toes not a<}Wi»te^ "mateh" the reform^reated individt^. then 
difference behveen these groups may simply refte:t difference in prior individual characteristics rather than dlfferen<»s due to the 
program. For exampte. the companson group may o^rr^>ri^n! men or mmorittes, skwif^ the outcome infom^on. 

If neected informatton is not ojltected consrstwrt with the da!a gathcrii^ prMCss tt«t is part of the research tfesign. the reliabHrfy of 
this information is in question. For example, if fottow-up information ^ to descrit^ the postprogram mitcom^ of all part^ipants m the 
reform, but information is ^thered only on thcw who found jote follwmg fl^ program, or if foUow-up fnformation is c(^ected on^ 
on those ^siest to contact. H wifl be very difffeuft to comf«nsate for the bm^ cr^Jted. 

Methods ot analysis must afH)ropriate to the pur|K>M of tl^ e^uattoa the kind of informatkm desir«l from it, and the nature of ttm 
data gathered, tf ttie methtafe are mrt a^H>fOprate ws-a-ws tf^se criteria, or if apprc^nate n^tt^KJs are not appfi^ corrK:tfy. the 
validity of the results will te jeofardlzed. For eacampte, "content analysis" may be the appr(«)rtate n^od for obtaining informa- 
. hon als)ut staff attitudes toward clients m new j^rsonnel r^pt^bHhte in ttw new reform, if qu^tionnaires do not used ^led re* 
spor^. If such respwses are subsequently recl^if^ into scates afl€r-the*f£B:t. in order to m statrrtical analyst technique, 
unkm»Am bi^^ may be introduce. 
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Summryof 

IntefprftatkHi 
Rndmgs 



In^roWngthe 
E^uat^ Program 



The answers do f\(A ex^t in a vacuum. Thtir cofitcxt Is the entire evaluation prcK^. in reporiinfl ttiem. it is necessary to anttcrpata 
ttre condHions ttia! must be imp<^ upon their mterpretaticm. 

In drawing cwcli«lon$ frwi evaluation findings, ttte sources, nature, and ejdent of Was must be ccMisKtered. Some ot the biases 
introduced have b^n elimin^ed or minimccd by studying certein asp«:ts ot prt^ram implementation and comf^nsating tfirough 
t?w use of stattsfical techniques. Ho*wer, m evalu^n j^rtect Ccmcli»«>ns. therefore, iwd to t» qualifted so that the user can 
wskB a judgnwt atM the reliaKtity, validity, and gwwaftzabilfty of m mkirmation prodiKrcd, and tf^ rejrficabitity of the evaluation m 
another program settmg. For example, if tte sample siZKs imed to study client subp«v«latk)ns in the reform turn out to be tw> small to 
yield tri^twotthy inforn^n — for criUcal groujs such as single parents \wth preohoo} vs. school age children, for mstance — then 
conclusions simply should not be drawn abmjt ^tse subgroups. 

Researchers are not wdHWiiy ex(»ct«i to make iwlicy or prt^ram rwommendaticms on ttieir stud«s. However, sociai 
program evaluation is not a sctentif ic effort, ite resute are exp^led to lead to son^ Und of action Just as the thrust of 
e^luatkm acthrittes is to stiiy the effects of plann^ ch^. evaluation informattcm is to ^ to p<«ftive action. Con^ently the 
; evaJi^tton contract fr^uently require ih^ pi^lcy nscommendatitms n^. Tte progression from mtuatm fmdmgsto condu- 
m^to rec(mnm^im>n$f(^m^i% a sensittve one. ptrtenKal b^ses pre»Ttjf^ a ct^ter^ ai e^h step. For exampte, the 
evriuation of tte new welfare retorm program may find that ^wt-twm w^re rec^nte ber^lted more from reform itm chronic 
users. This does not nece^ri^ justify Imping to ttie corwius^n that tf« program d£«s not work for tong-lenn recipiente of inconvi 
subsidies, and the recommendation that servm be targeted more exclusive^ to short-term welfare consun^rs. One would hm to 
addre^ issi^s such as the preprc^ram charm^teristii^ of lhe» two grmjps and the differential effect of aHemative servK^e mixes and 
^i^nces on tl^r outcome. 
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succession of political and organizational actors with differcnt perspec- 
tives and investments. Each program, however, can be undcrsiood in 
terms of a number of common attributes that reveal assumptions about 
( 1 ) the nature of the problem the program is to address. (2) who and what 
should be changed, and (3) what types of changes are needed and how 
they arc to be accomplished, i.e.. the "thetjry of change" underlying the 
design of the program. 

Each social program involves a variety of strategies for accomplishing 
change. The strategy that lends to receive the most public policy 
aneniion is the program's formal sen ice intenvntums directed to 
}»uiicipants. TTicse involve particular mixes and sequences of serv- 
ices — and sometimes subsidies— that respond to a particular problem 
and distingui.sh a given program from others. Less visible to ihc public, 
lut just as important to study, is the program's implenwutation sfrarefix, 
or the organizational features of a program that are expected to provide 
an tnvironment in which the service interventions can have maximum 
effect. This strategy rcprcsenis an additional change agent, or set of 
organizationul imenentions that can influence the extent to which the 
desired changes arc accomplished. These implementation characteris- 
tics arc ( I ) the structural and functional aspects of the organizations 
responsible for a program, and (2) the nature of the serv ice delivery 
systems operated by these organizations. 

Service delivery sy.stcms expose the client to the program's service 
interventions, facilitating a satisfactory exchange between program 
eflbrt and the social benefits associated with change. The way this 
exchange occurs is greatly influenced by the organizational context in 
which it takes place. In turn, this context has a significant impact on the 
program experiences of clients and client outcomes. Therefore, process 
evaluations that focus on how effectively and efficiently a program's 
organizational system worics. and what influences it has on program 
outcomes, provide essential information. Studying implementation 
poses unique measurement dilemmas, however. Some aspects of organ- 
izational life cannot be defined in a form that lends itself to statistical 
analy sis. Other methods can be used to summarize and draw inferences 
from .such information, yielding u.seful insights abtjut these lesser- 
known, but critical organizational interventions. 
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Oulconic evaluations, on Ihe other hanti focus mor: exclusively on re- 
Imionsihips among the characteristics of clients entering a M^rvice deliv- 
ery system that arc thought to affect their outcomes, the program's 
service interventions targeted to these clients, and tisese clients' out- 
comes. Outcome evaluations typically seek informatiim on the net 
effects of scn'ices on outcomes, requiring that the evaluator trade 
information scope for information accuracy. 

Sometimes important measurement issues must be neglected in out- 
come studies, such as the precision of the definitions that distinguish one 
kind of scrvicT activity from another, the development of profiles that 
describe the tnost frequent conllf.urations of services and sequences 
deHvelT^d: the capture of information on significant attributes of the 
service imerventions. such as length, and the gathering of information on 
ini(K?rtant characteristics of client outcomes, such as quality, magnitude, 
and duration. Furthermore, outcome evaluations have often been limited 
to tlK program's elTccts for j^nicifntnis at the point of program c^omple- 
lion, neglecting longui-icrm impacts. And some outcome studies have 
ignored the program's impact on other key individuals and entities, such 
as employers, educators, or other social programs, 

Neverthel/ss, studying outcomes is the major oversight responsibility 
in judging the value of social programs If inttcome evaluations are to 
provide information genuinely useful for fX)licy, planning, ami manage- 
ment purposes, the goal is to define and measure as inclusive a range of 
important interventions and outcomes as a feasible reseaa^h plan per- 
mits. Process studies can then offer information rcgmTding why and how 
the observed outcomes o:cur. Comprehensive program evaluations, 
therefore, will address both implementation and outcome issues, afford- 
ing users complementary information about the dilTerent influences and 
effects that describe a particular program. 

Again, using the work/welfare demonstration project as an example, 
chart 1.5 helps us sort out the different influences and effects studied in 
process vs. outcome evaluations. 

Pressure to justify a prognun's costs by demimsiraiing its sixrial utility 
has often encouraged resistance to cf>ndutling process and outcome 
evaluations simultaneously. Outcome evaluations represent an ineffi- 
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cicm use of it?s4)UR*eH. however, if carrieU out in the abwncc of puKcss 
cvaluationn thai can help explain pmgram etTccis* Hie goal of outcome 
evaluations—to estimate program impact — and the goal of prwrens 
evaluations— >io undet^itand why such elTecls wca* produced and how 
i^rvtcc or implementation strategies can be mixiified to moa^ fully and 
cfficjem!y produce desired outcomes— are not in ccmflici. The achieve- 
ment of ht)lh goals assures moa* ctmiprchcnsivc. complementary infor- 
mation for decisionmaking (Judd 1987). 

The Characteristics of Evaluation Approaches and Methodologies 

In seeking answers to complicated questions about pmgram implem- 
entation and impact, the ultimate objective is to come as close as possible 
to establishing which influences arc cuusinfi certain effects. Was it good 
case management or the formal service received that resulted in a better 
outcome? Untangling cause and erfcci is a difficult task. Science can 
give us accurate estimates but noi certainty. And it can only cifTer greater 
precision if we follow scientific principles and methods very closely. 
Sometimes that is not pi>ssible. I'he questions lead us clsewhea\ or there 
arc organizational or political reasons why we cannot use a particular 
evaluation strategy. Not all of our evaluation options pmvide us with 
information on cause and effect. They may. however, offer extremely 
useful information of a different kind about the relationships in which we 
are interested. 

One basis for differentiating among evaluation strategies is the extent 
to which different research (icsi}ins produce information that explains 
cause effet f relationships, as compared with determining associations or 
correlanons among influences (short of establishing cause and effect), 
or simply providing important clues or insights about the possible nature 
of such relationships. Following scientific principles and methods most 
closely, the ideal experiment gives us the most "unbiased"' information 
about pmgrams. In realistic social program settings, however, not even 
experimenlal designs can paxluce infonnation miracles. Therefore, we 
need to apprtciate the spectrum of design possibilities available to us. 
One way to 'describe this spectrum is found in chart 1.6. 



ERIC 



Ctl8lt1.S 

tettowBN 8telfetf itt ftpmt awl ftitwut fwlttgtfwi 

Tftr^ firmijm Eq^ctttf to Rtcslvt fidptm tatemirtions 

The Proflrani's Parttcipants PubHc AisiBtsnce Recfpmts Otf^r Entitres Exited lo Benefit irom Procram 

Key Cfwradenstics Other Welfare Pn^rams. 

• Age • Other tramjng and employment pn^rams 

• Sex • Unemptoyment insurance program 

• Education • Other educational programs 

• Marrtal status • Health programs 

• Number of Lhildren • Economic development programs 

• Work history 

• Income htstofy 

• Frequency/duratfon of periods on public assfstance 

Desired Oittcomss of Program 



For Program Partcipa.ils 

• Increased employatNlity 

• Increased occupationalsWIs 

• More durat)le employmem 

• SiiflfCtent earnings to reduce or eliminate the need for a welfare grant 

• Greater family stabiitty^irfe qualrty 



For Other Entities 

• Reduced costs 

• Reduced ^rvfce demands 

• Greater coordination and less duplication of effort 

• Increased tax revenues 

• Increase job creat^tm 



Interventions Targeted to Participants 

Emptoyment/ Servic«s m 

Trammg 
Services 



Supprmot 

EmploymenlT f 3ini?T(j 



f^^fil t^iet Intermtim Eqi^il to Oe &lred O^t^mes 

Interventions Targeted to Other Entities 
To Employers 



Educal^mai occupy] iona) 
^k'i^ resting 

^t) seanrft ass-jstancf 
9ni placement 



He3*!rf caff 



Subsidies Assoctaled 
^ Emptoyrneiii' 

• BaSK, ncomf tampon 



and sow ti^ 



To Other Pfograms 



ProgrBm bnpitnraiitatloii F^tom Eqn^ed to Foster D^retl tk^mn 

OiB^Bf tettcs of 0ie Wsttarv Retonn Prograin s 



Gtim€t9risti«s of Weifsrt R«f^ Arogmm As an Orimtzatl^i Syttam 



Structural Characteristics 
Stale Executive Committee 



State AdvisoiV Committee 

State Program Adminrstration Structure 

Local Program Management Structure 

Public/Private Service Subcontractor Systems at 
the Community Level 

Community Councils 



Functional Chafidenstics 

• Acquisition of funding 

• fte*ds ass^mem/resource atotion 

• Formutetton of pfx^ram fwlicies 

• Coordii^tion with related programs 

• ^ogram accountability: plan rwew. 
program monitoring, pr(^ram evaluatJon 

Provision of expert opinion to ExKutive 
Committee 

Central administration, planning, management, 
mc^torif^i^fting 

Local administration, piannmg. contractmg. 
program operation, mcmitonng and repofting 

Sui^rvslon of h^i ^ice deiiveiy systems and 
provsio^^ofserwes 

Provision of gr^sroots fe^back from kol ci^rts 
and servkje providers to tt>e tocai organtetional 



Service Deftvery System 

ftjlici^ and Practices 

• Outreach 

• Intate/eligibifity determinatfon 

• Appraisal/seivicc plsnmng 

• Service assignment 

• C^managen^nt 

• Sufwision; 

m ^ucational/1rainmg/'emplo>'ment sites 

• Referral to otlier programs/alternatives 
^ Placemem 

• Foftow-up 



^omnwy of tesM bwM^b^piiM to ft^M<tt ft> OvtcoiBe EvstestNiQt 
Sl^yit^ Pnq^ topfa»m Stetj^tHI PfCfnnn tetom ei 

The focus is cm relationships among M organizational elements of a The focis is on retefiwiships l>etweeft the interventfons targeted to prc^jram 

progr^ and the potential inf lutfif^ of tt»se organizatronai factors on participants and the outcomes of tl^ program for participants and other 
program outcomes. entiti^. 
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The characteristics of evaluation ahematives relevant to information 
about cause and effect are not the only attributes that distinguish one 
evaluation option from anotlKr. C^c must considerother characteristics, 
such as the rigor of ih: resmrch design and the realism and ^eneralizabiUty 
of the information different kinds of evaluations provide/ Each alterna- 
tive may vary in terms of these attrilmtcs. depending on the decisions 
matte in developing the research design guiding an evaluation and the 
way this methodology is actually implemented. For example, for each 
type of design described in chart 1 .6, there is a variety of methodologi- 
cal choices that differ in the extent to which they produce accurate infor- 
mation — information that realistically reflects a program's natural state 
and can be generalized beycmd the individuals or other entities studied. 

Even the ideal experimental design has its own set of tradeoffs in this 
respect. It is the most rigorous in the scientific sense, in that it provides 
maximum control over certain kinds of bias. However, control is 
achieved by randomly assigning eligible clients to served vs. nonserved 
groups. This essentially changes the program's expected implementa- 
tion strategy, which calls for service assignment at the discretion of 
professional staff. 

Even if staff do not subvert random assignment, and client advwacy 
groups do not jiemand that all eligiblcs be served, we still find ourselves 
asking, "Hovi, did random assignment itself affect program outcomes?" 
Or if, for a variety of reasons, the experimental sites for studying a 
program are not repre.sentalive of the areas to which evaluation results 
are to be generalized, the informauon obtained will be of less value no 
matter how unbiased it is. 

Experimental designs also place limits on the variables and relation- 
ships that can be studied, in order to institute control, which then leads 
us to wonder whether we have studied t^ most important influcntrs and 
effects or missed something critical. In general, experimental designs are 
stronger with respect to rigor, or the ability to isolate certain of the effects 
of a program's interventions, but they tend to be weaker on realism, and 
often on generalizability (Blalock 1964). 

Quasi-experimental designs interfere much less with a program's 
service delivery practices, but they produce information of varying 



CHartl.S 
Rtmrch Dtsigns 



N0imxp9Jlfn«f?tel 

ExplDratory 



Descriptive 



Ouasiexpcriwntdl 



To e)qp^m relafionsfiips among imptementat«»i features, 
service interventions, outcon^. and/or tlm program's envi- 
ronrmnl. m orcter to rdenfify what factors operating in th^ 
relationships imy be the most imi^rtant ones to stm^. and to 
ot^tain insets at)Out ttre nMure of their influent f more 
systemat^ stutfy. 

To descnt^ reiatk>n^ps among any or all of tt^ atiove. 
focusing on influences and effects known or considered to t)e 
probable key t^ors. to detemiine the extent to whicii they are 
interreiated — i.e.. a^jciated stat«ticaJfy. or cm^wd — 
short of attempted to ^tatli^ cause-eff^. 

To estifMe (^i^e-efkct reiaiionshtps t>etween a program s 
tnterventitms and oirtcom^, ccrniparing program participants' 
outcomes with a $tati$tfCBtty amstrwted compmst^ grat^ of 
individuate who have not receive the programs' interventior^. 




El^frline^l DaslgflS To mw prec^ <^tmmme cause^ffe^ reiatimishfps t)^ween 
mten«rTtt(ms and otrtC(Hros. comi^ring ^e outt^m^ of partici- 
pant who received ti« program's interventions those who 
havenotmc8ivedtiiem,t}^}onth6 TBmkHms&gmm^^z 
fK)crt of cfcnts to a "treatwT group arni a '^on-tr^twl cwrtno^ 
gm^ pm to the intn^luctkKi of the interventions. 



Eiamples 



ParlfC^mt'Ob^rvatfomnd case studies ranging from mformai 
to quite formalized studies 



Survey or paiwl studies of program participants and rtJrer 
mdviduais to be changed by the program, ranging from 
relatively unstructured to highty soph^cated studms. 
example is (f^fwtw {^t^ ^Hstcom studm that Invc^ 
simpfe statistical analy^ of program foltow-up information. 



tttfemtmi gross autcms ^ih^ which compare ^ 
outconm of particuiar »ibgroui» of program participants who 
have im\ assigned differ^ interventions, using one group as 
a comparisfm gn^ior anottier. adjusting statrsticaiiy ^ 
seta^tkm bias, and ^>ntroiting for Influences on the {^-ogram's 
environiT^nt tt^ could affect outcome. 

Net fmpact ^tf(Ses.i^ki\ compane ^ outccms of program 
partiapants Mrith tt^ of a grcMip of indivk&tals ^ simitar to 
tf^ partlcfpantsa^ possible ^ wrtio have not reived tite 
interraitions. ^atisBctf Ry accosting for setectton bias, aKf 
c^t^WfHI for the j^t^nrnt's environmmtt. 

Fl^ mipmnen^^fA^hamparB theoutom^c^^lre^ed'vs. 
"nomtr^t^** groui^ th^ are equivatenm tBxm of prepro- 
gram ctoractfstics that n^ affect outcmnes. controlling 
stettsttofly tor infltwnces in the program's environment. 
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accuracy depending on the methods used to control for bias. Again, the 
scope of an evaluation is neceHsarily limited, but Iheix: is greater flexibil- 
ity in studying programs as they actually operate. If car(!funy developed* 
these designs can achieve the desired generalizability in terms of iden- 
tifying program outcomes across diverse sites, TTierefore, quasi-experi- 
mental designs arc generally weaker in controlling bias than exf^rimen- 
tal designs* but frequently stronger with rcspect to realism and general- 
izability (Campbell 1%3). 

Exploratory and descriptive designs clearly do not offer as much 
control over biases as the two designs just discussed. Assuring that the 
information from surveys, panel studies, and case studies has been 
purged of subjective judgments is much more difficult. Within this 
constraint, however, the sampling, data collection, and data analysis 
methods incorporated within descriptive designs can vary from rela- 
tively unsophisticated to highly sophisticated. 

Well-struclut^d exploi^ory designs can also yield reliable informa- 
tion. Far more flexible in what can be studied, the more rigorous 
exploratory and descriptive designs can reflect the true complexities of 
programs in their natuml settings better than other methodologies. If 
sampling stmtegies are adequate, information obtained using descriptive 
designs is very gcneralizable, competent survey research being a case in 
point. And these designs are appropriate for .studying both implementa- 
tion and outcomes (Davis 1967). However, they represent weaker 
methodologies in terms of rigor. 

These tradeoffs among accuracy, realism, and generalizability need to 
be viewed positively, as providing a framework within which different 
compromises can be ci^fied in the context of other tradeoffs, particularly 
organizational and political ones. As we try to incrca.se our proximity to 
the truth, we should appreciate tte different tradeoffs and options 
available and make the best choice given the many demands and 
{Mi^ssures surrouiKling that choice. 

The Evaluation Approaches Emphasised in thh Book 

The three noncxpcrimental evaluation approaches selected for atten- 
tion in this book represent only one set of choices. They were chosen 
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because ( 1 ) they resixJiKl to the organizational and political realities of 
ongoing social programs; (2) they provide the least-biased information 
short of imposing random assigjiment proccdun^ on service assignment 
policies; (3) they represent an integrated and comprehensive approach 
to studying progiams over their planning cycles; ami (4) they are 
oriented to the practical decisionmaking needs of those rcsfKmsiblc for 
the success of such programs. 

Net impact Evaluation 

Tte approach used in determining the net impact of programs, or their 
**retum on the public investment/' is based on the principle that this return 
cannot be estimated precisely without comparing the outcomes of 
participants with those of similar individuals who have not received the 
program's interventions. TTiercfore, the selection and measurement of an 
appropriate comparison group are key tasks. In this context, cause-efTect 
relationships between interventions and outcomes for men vs, women are 
studied^ using a rigorous quasi-experimental design. The influence of the 
length of time a client is exix)sed to a particular service intervention and 
the effects of local program environments arc considered, focusing cm the 
magnitude and duration of the outcomes achieved. A comparison group 
strategy utilizes regularly collected administrative data and adjusts for 
potential nonequi valence in the "treated"' vs, "nontreated'' groups through 
Oie applicaticm of statistical techniques. 

Because of the rigorous research design required to estimate net 
program impact, such evaluations are necessarily more limited in the 
number and kinds of questions ll^y can answer. However, they afford 
the best means of establishing what exclusive contribution the fHogram 
is making to the short-term and longer-term fiscal and programmatic 
outcomes observed* as compared with other fK)tential influences that 
could explain these outcomes. 

Gmss Outcome Evaluation 

Hie approach used in studying a program's gross outcomes is a 
twofold strategy for gaining an understanding of a broader range of 
outcomes than those that can be studied in net impact evaluations and 
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acquiring marc in-dcpth informalion about the intensity, quality, and 
durability of such outcomes for client subgroups receiving diffeami 
program interventions. TTiis approach can make optimal use of and build 
upon a program *s automatixl information system. 

The first strategy utilizes a descriptive research design lo study 
n^Iationnhips between service interventions and gmss client ouu omes, 
as well as gross outcomes for othere to be benefited by the program. The 
^ond strategy, referrcd to as a Jiffirentiai f^ross ouh omc anuixsis. uses 
a quasi-experimental design to analyze fH>tcnlial cause-effect relation- 
ships between program services and outcomes, sorted by types of 
interventions and kinds of clients. This approach does not involve the use 
of a comparison group of nontieated individuals* It relies, instead, on 
using one treated group as a comparison for anothen This strategy also 
involves a study of service assignment practices and uses statistical 
techniques to compensate for lection biases. In this sense, the approach 
combines careful outcome evaluation with a complementary study of 
particular aspects of pmgram implementation. 

Frui^ess Evaluation 

The approach taken in stud) ing program implementation applies basic 
systems analysis concepts to social programs that are viewed as organ- 
izational sysrems (Mintzberg 1979). Emphasis is on how efficiently and 
effectively the p^rts of this system interrelate to achieve system goals, 
and on the influence the articulation among parts has on program 
outcomes. In evaluating implementation, the system is classified into its 
com|>onent parts and their integration with one another is examined. 
However, no matter how we!! such a system may work internally, it is 
also shaped by its relationships within a larger environment, including its 
funders, resource allocators, auditors, other human service organizations 
with which it is expected to interact, and key actors in its home 
community. Therefore, these relationships are also sub^ted to scrutiny. 

The assumption underlying this approach— using either an explora- 
loiy or descriptive research design— is that organizations and systems of 
interrelated organizations are goal-directed, and successful g(^l achieve- 
ment is a function of how well organizational structures and functions 
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work together. Information about these iniporiant oi?anizaiional link- 
ages can help explain why programs pit)viding similar interventions to 
comparable clients can produce quite diffen?ni outcomes. More often 
than not, the success of a program in conforming to its intent is more 
dependent on organizational factors than on the influences typically 
given maximum attention in the public ix>Iicy arena. Such information 
is essential in making useful program mcKiifications (Williams 1976). 

The Organiauition of the Book 

The bot>k is organized to express the particular concept of evaluation 
outlined earlier This concept views evaluation as a set of complemen- 
tary appit>aches that can offer practical information on pn>gram implem- 
entation and impact for state and local decisionmaking. Evaluation is 
also seen as requiring a special fiscal organizational, and political 
capability, which can benefit the state and local agencies operating social 
programs. 

Chapters 1 through 4 focus on research appn^aches and methods for 
carrying out net impact, gross outcome, and process evaluations, respec 
lively. Each set of approaches and methods has its own unique explana- 
tory advantages in providing information useful in improving programs. 

Chapter 5 encourages tte reader to treat the development of a user- 
focused evaluation capability as an opportunity to increase staff skills 
and expand pix>gram resources. The author discusses practical planning 
challenges posed by evaluations coiKiuct^ at state and local levels, and 
suggests realistic alternatives for acquiring the funds, expertise, and 
other resources essential to an evaluation effort. 

The final chapter builds on the issues discussed in chapter 5 by 
proposing ways to stit^ngtten evaluation capability and the use of results* 
and cxmcludes with a brief overview of il^ book*s major ideas. 

Each of the chaptcni utilizes the same basic format, whose purpose is 
to assi.si the reader in absorbing complex material by moving from the 
general to the specific. This pitjgression begins with an oveniew of the 
chapter^ main ideas, which are relevant to the evaluation of a Iwioad 
variety of scKial programs and detnonstration project This overview is 
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followed by an applk utum of ihese ideas lo a contemporary cave 
example, JTPA, Each chapter concludes with a brief exploration of how 
the ideas? in the chapter rW^i/c to and complement ideas in other chapters, 
and a summary of the main iX)intH to be retained by the reader. 

In the three technical chapters covering reseanrh strategies, the over- 
view in some cases gives the a*ader a sense of the state-of-the-art 
regarding the evaluation apprx^ach profKised. Where feasible, the appli- 
cation of principles and methods to the JTPA example are pa*senied in 
a manner consistent with the steps in the research process outlined 
earlier. Specific illustrations within these chapters anc related to the kinds 
of pragmatic program decisions made by fx^l icy makers, administrators, 
planners, managers, or professionals responsible for pn^gram oversight 
There is alsi> a commitment to expressing technical concepts in practical 
language. 

In conclusion, it is important to emphasize that looking at the political, 
organizational and technical asi^^ects of evaluation recognizes their 
influence in puxlucing objective information for decisions affecting 
program evolution. The information user, evaluation funder. poli- 
cymaker, program professional, and e valuator must work together as a 
novel partnership to guarantee that the infomiation product justifies its 
effort and cost. This perspective on pn^gnim evaluation, stressed 
throughout the book, acknowledges the reality that researeh techniques 
are only one major part of a larger system of influences ihat shape the 
form and content of program decisions. 

NOTES 

I . TV dtrcaik of the ! M7(K \^itnevscd a series olbrgr-wak msiu>n*tl human mth iccrikTntmvmttitm 
projcris. follimtHJ by Mihviantial eviiluatitmv «if their cHectv. These Hca* innovaiive j>a>jci:is 
desfgitertl lu te>! allcfiuitivc pfi)|>iafn Mjalr^nrs whh purtiLubtr clnmi gmup^. Hxampltrs arv ihi* 
extensive Seaitttr/fVnver Im^nne Mjinfenamr Hxperimems. ihc Cancr adminKtnition uclfarL* 
reform pmjeitv, ttw finir-Maltr Fkr xihkr InierjiinTmnHTnal Gnint Project, and CETA. Ai the same 
lime, ifwnp a nwinher of mr! impnti smftcs of in<?re olahlmhed pn>gninis. •tmrh as \\^ Wiwk 
Incemive Ptngrani and Head Stan. Mom evutokniN etm-enlrafed on a-luiu-mship heiut-en 
Kerviee mlervwitons and elien! twicomcs. but dei'a(kr marked the hefrmnuig of imeres! in fhe 
innuencc of progncn imf^emeniatitm as Hell Sisrial R&D ere at iheir /eniih. iSce Mirenpotl. 
Bm>«m . and Ciuenm Ah examples «)f majiH* e\ aiuations if i the I V7IK ami 1 9H(K. ) 
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2. Tlie volutms* and ibtir auitMjpi air liMcd in the a}^*ndix. Am^t^r scnrs ut cvaluaiU^n guichrii 
that tnay be of intcrrst ts^sel i^f xhvec volume> Un Mudy mg Hval pn^rdm** under CETA. jw|wed 
for Re|SH^ I. Hmployimn! ami Training AdminiAtmihin. \ S. Def^n^t vf l^bkn in ! v?K iSum 
et ai. 197¥>). Sage Publk'aiKwui has ahin jroifuced a wncv of nine mtwn)graph«» on bixsk research 
appn^'bei^ and methiHls tilled ihe Program Eya^muon (Hennan 

3. TTu? {ireiaiujv a*vicw in cha^^er 2 in also an cu'cllcni MHiac of mrimmiuin ahi»ut ihe baxiv kw* 
developing performance stand;ird«i imficau>ni m employment and training. 

4. Tt^ ihicc itaiici/cd terms, and the di^u^Mon uf iraifcolls amimg iheve v'hufiKten*»lKA ot 
fTveanirh. are basixi on an inleawiing aliiie by I jf*.lie K»%h. a ueM-knn\wn vurvrv r^Mrarvh expert, 
titled "RepnM'ntaiK^n. Ram!i>nti/atH^n. am! C\>n!r<?r" \ I**75). 
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Part II 

Complementary Approaches 

to 

Evaluating Programs 



The comprehensive concepl of evaiuaiiim profK>sed in Part 
I gives afiention u> \\k imponani rclaiionshtp between tech- 
nical lesearch approaches and the organizational and poHiicai 
context of state and iocal pn^gram evaluation. 

The emphasis in this section is on the critical research 
aspects of evaluation and the significance of prtKiucing 
complementary information about program implementation, 
outcomes, and impact over a given program planning cycle. 
Of particular concern is the scientific quality and comparabil- 
ity of infomiaiion produced by evaluations. 

The ihf.e chapters follow a common chronology and 
address the same set of re^^arch issues. However, the chapters 
vary with respect to which research issues are given primary 
attention. This reflects the different research challenges in- 
volved in studying diverse aspects of programs, as well as the 
contributions different socirl science disciplines make to 
pmgram evaluation. 
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Evaluating Net Program Impact 

Terry R. Johnson 
Battel le Human Affairs Research Centers 
and 

Ernst W. Stromsdorfer 
Washington State University 

There are mwix mmmU sm ial seififi\;s in whU h ffw restwrher i an 
imnhiinr something Uke experimental dcsi}in . . . whieh rfuikes a true 
eMfnrinienf possible , . . Such siiuatiom can he re^ardrd as t/uusi- 
experinmuai t^esi}:n^, fWe wed loj ennmra^e the lailizatiim 
siii h ifuusi-experimcnfs and lo im rease anareness af the kinds of 
seniujts in w hich (^pormnines to erttphy ihem mrur. 

OomXiX C ampheH and Julian Stanley 

Hxperimeniui and QmmLxpvrimenUtl 
Dt sisim f(» Hcsean h 

iieneru\ Concepts and Methods 

In a world of scaa^e rcsouavs and gn)wing federal deficits, it is 
increasingly imponani for scKial programs lo diKument the anum that 
society as a whole receives on its investment. To effectively compete for 
limited fedenil, state, and local rcsourcres, social programs that have 
provided services for many years find themselves having to document 
clearly the benefits and costs of their services^ In addition, innovative 
demonstration projects must properly document their outcomes and 
Qosls to company ^^ ith existing social programs or allcmali ves to meet the 
same goals, so policymakers can make informed decisions on whether 
the demonstration should be continued in its current fomi, expanded or 
reduced in scope* or discontinued. The emphasis on social program 
accountability applies to ail types of social programs, including those 
that provide l^ahh, housing, or inccmie maintenance services to tiK: aged 
or to children, and has been an integral part of the fecteral employment 
and training system for economically disadvantaged youths and adults 
for the past few <fccacfcs. 

43 
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The iitroniiation rcquianl lo make a decision on whether to exf^d. 
eontracL or discontinue a social program begins with a net impuvt 
evaltHitum. Numemus factors affect the outcomes of panicii^nis served 
by any social progiam. TTiese factors include the participants' own 
characteristics, the program environment, and the services provided. 
Recognizing that many factors may affect prognuii outcomes, a nci 
impact evaluation attempts to answer the fundamental question, what 
difference does the program make? This is in contrast to a uross impuvt 
evaiiiutHm, which focuses on w hcther the outcomes of interest for par- 
ticipants arc greater after the program than before the program, and does 
not deiemiine whether the program services per se caused the observed 
ditTerences in outcomes. 

A net impact evaluation examines the changes in outcomes from 
before to after receipt of stKial services and. in particular, examines 
whether imy change can he causally attributed to the services received. 
By comparing the outcomes for participants of a given social program 
with the outcomes that would have ixxunvd if the program did not exist, 
a net impact evaluation provides valuable information on program 
benefits that can be combined with cost information to make informed 
judgments about the cost-eflcctiveness of the prx>gram. In this chapter, 
we illustrate a general approach to assessing the net impact of a social 
program by describing how to estimate the net impact of employment 
and training programs for economically disadvantaged individuals on 
piirticipants' post program labor market experiences. 

The details of the net impact evaluation strategy desi-ribed in this 
chapter have been tailored to a specific social program, but the key 
elements of the net impact approach have general applicability lo other 
social programs. For example, the first element of any net impact 
analysis involves the development of an appropriate etmcepUiul frame- 
work. This framework identifies the key research questions addressed, 
the outcomes examined based on iho^ questions, and the participant 
subgmups and program services delivered to clients. 

The outcome measures examined constitute the dependem variables 
for the net impact analysis. The ciniceptual framework specifies the key 
relationships investigated between the dependent variables and meas- 
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ures of various causative factors, such as participimt characteristics, the 
economic environment, and the program services. These factors that 
affect the dependent variables are called independent variables. By fully 
specifying the net impact questions of interest, the conceptual frame- 
woric identifies the key relationships between the dependent and inde- 
pendent variables to be analyzed* 

The ?«rcond component of a net impact imalysis is the development of 
a research desif^n that provides valid answers to the questions of inteirst. 
To select an appropriate rcsearch design for a particular social program, 
one should be guide<i by several criteria. These include internal \xiiidit}\ 
external validity, statistical ejftciepwy, and feasibility. Internal validity 
refers to the ability of the design fo yield unbiased estimates of the causal 
relationship between program services and outcomes: that is, valid 
estimates of the net impact of the program. External validity refers to the 
ability of the research design to achieve results that can be generalized 
to a broader population than the specific samples of individuals upon 
which the analyses arc based. Statistical efficiency relates to the overall 
sampling strategy and the need to utilize sufficiently large samples to 
obtain prccise answers to the research questions of interest. Finally, 
feasibility rclates to the ability to implement the research design and 
obtain meaningful results in a timely fashion and within the limited 
lesc^unres available. 

Although all of these criteria are imfwrlant considerations in selecting 
a research design, the criterion that receives the most attention in the 
research literaturcis internal validity, or obtaining unbiased estimates of 
program impacts. To meet the internal validity criterion, one must be 
able to comparc the observed outcomes for participants with the out- 
comes these individuals would have achieved in the absence of the 
program. To do so. one needs a comparable group of nonparticipanls to 
serve as a comparison group. 

The key to internal validity is the comparability of the groups being 
compared. One can obtain valid causal infercnces about the net impact 
of tlie program only if other factors that affect the outcomes are statisti- 
cally equal for the two groups. The most effective way to ensurc this in 
a .social pmgram evaluation is to use a classical ex{Krimental design in 
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which individuals eligible for program services an? randomly assigned to 
treatment or control status. By randomly assigning some individuals to 
receive services and other cligibles to not receive services, the two 
groups should be essentially identical on all dimensions that might affect 
program outcomes. Thun, any observed differences in ouicomes can be 
reliably attributed to the treatment rather ihan to pre-existing differences 
in the characteristics of the two groups. 

Although this method has often been used to test the effectiveness of 
new service interventions, it is not generally used to evaluate the net 
impacts of social programs.^ This is, in large part* because of ethical, 
legal* and programmatic concerns. As a result, a matched comparison 
group approach^ — referred to as a quasi-experimenfal desij^n — is typi- 
cally used. Such an approach relics on some matching rule based on 
measured characteristics- A matched comparison design could involve 
the matching of service providers, in which one provider offered the 
treatment and a comparable provider did not; the matching of individu- 
als* in which participants within a given program are matched to 
comparable nonparticipanis; or the design could include elements of 
both. Because the feasibility criterion in many cases eliminates the use 
of an experimental design, in this chapter we describe a quasi-experi- 
mental net impact evaluation approach. This approach should yield a 
considerable amount of useful information on the effectiveness of a 
social program. 

A third element of a net impact analysis is the dafu collevtion and 
analysis plan. A guiding principle for collecting data is to make sure that 
the information used is measured comparably for both participants and 
comparison group members. Thus, if certain data items are obtained 
from different social service agencies for the two groups, one must 
review the data collection forms and procedures to determine compara- 
bility. In addition, it is particularly imjKJrtanl in a quasi-experimental 
evaluaticHi to obtain extensive historical informauon on a!l key outcomes 
of interest to detemiine whether the participants and comparison groups 
are similar on these measures before participants receive program 
services. If there arc major differences in outcomes between the two 
groups before program participation, this makes it more difficult to 
isolate the true efiects of program services. 
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Once the data arc collected, the next step is analysis. Because tte 
genera! statistical i^>sues inherent in most i^i impact analyses are quite 
similar, the plan for analyzing tin? data collected for evaluatiras of 
different social programs includes many common elements. For ex-^ 
ample, a key element of any analysis plan is a detailed examination of the 
likely extent of selection bias. Selection bias ivfens to tl^ potential non- 
comparability of the i^rticipant and comparison groups due to the fact 
that individuals self-select themselves to become (mrticipants. In addi- 
tion, the analysis pism must include a strategy for estimating the net 
imi^ts of the program overall, and for major program services and 
participant subgroups to answer the questions identified in the conrcp- 
tual framework. 

Finally, although the results of a valid net impac t analysis can provide 
useful information on the extent to which participants arc better off as a 
result of receiving program services, Ihcy are not sufficient to directly 
address questions related to whether the program should continue to 
operate at the same IcveK be expanded, contracted, or discontinued. To 
determine whether a social program is an efficient use of public re- 
sources, one must sum the benefits to participants and the benefits that 
may accrue to other segments of society, and compare the total benefits 
with the total costs of the program. That is, one mus» conduct a hcnefit- 
cost analysis. In practice, this involves measuring the oenefits to partici- 
pants, taxpayers, and government that accrue over time, properly 
discounting the benefit stream into current dollar values, and comparing 
the total benefits with the full social costs of the program. 

The remainder of this chapter illustrates the key elements outlined 
above of a quasi-experimental net impact evaluation of a social program 
with an application to employment and training programs for JTPA- 
eligible disadvantaged adults. Although the details of the net impact 
evaluation strategy described in this chapter are specific to employment 
and training programs, we emphasize the broader applicability of this 
general approach to other .social prognims. 
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Application of the General Concepts and Methods: JTPA 

The net impact evaluation slmtegy described here was denigned in 
renponsc to an increased need for Reliable evaluation information at the 
state and IiKal levels concerning the efl'ects of employment, education, 
and training programs funded under :hc Job Training Partnership Act 
(JTPA) of 1982. The JTPA significantly changed the employment and 
training system in a way that gave states and Iwal service delivery areas 
(SDAs) much greater responsibility for program accouniabiUty com- 
pared to the Comprehensive Employment and Training Act (CETA). 
This increased the need for reliable infomiation on program effective- 
ness at the state and local level at the same time that the federal 
government greatly reduced its role in providing such information. 

As a result, stales have less access to reliable fecteral-level data arni 
federally sponsored research to assist them in making informed judg- 
ments about program oversight and social policy. To fill the policy 
research needs of the states, this chapter describes step-by-step how a 
state can estimate the net impact of its JTPA programs on earnings and 
welfare dependency and pmvide valid information on the investment 
return from the JTPA. 

In developing a state-level net impact m<x!el of JTPA programs, we 
were guided by several considerations, Firsu for the model to assi.st states 
in meeting their new accountability responsibilities, it must produce 
scientifically valid estimates of the net impacts of JTPA program 
activities and services on relevant participant postprogram outcomes. 
Second, the model must provide meaninf^ul imd timely information that 
can be understood and used efficiently by a relatively nontechnical 
audience. Finally, the mo<tel must recognize the resource and other 
practical ctmsfraints that states and SDAs face. The two most important 
practical realities affecting the recommended approach are that ( 1 ) states 
and local SDAs will not generally be able to implement an experimental 
design in which eligible applicants are randomly assigi^d to treatment 
vs. control status, and ( 2) states and Uxal SDAs will not generally be able 
to conduct follow-up interviews with a large sample of participants and 
comparison group members. 
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Because ihe proposed net impact evaluation model rclies exclusively 
on administrative records {mm .^veral agencies that a>Ilect these data as 
part of the normal operating process, the model is widely usable. This 
model supports timely analysis conducted within the economic and 
political resource constraints faced by states and SDAs.^ At the same 
lime, ii must be recognized that this approach limits the questicms that can 
be addressed and the variables that can be used to adjust for various 
statistical problems that could threaten the validity of the analysis. Thus, 
in evaluating otN^r social programs that face different economic and 
political constraints, one should consider supplementing the researih 
design described here to include primary data collection activities. 

The remainder of this chapter describes a detailed approach for esti- 
mating the net impact of employment and training programs for disad- 
vantaged individuals. The chronology of the discussion is as follows: 

1 . We summarize the lessons learned from the research literature on 
employment and training net impact evaluations. 

2. We describe the key elements of the conceptual framework for a 
state-level JTPA net impact evaluation by building on previous 
studies. This conceptual framework includes a description of 
program outcome measures, the trainee subgroups for which im- 
pacts should be separately measu^^d, the program activities (serv- 
ices) to be examined, the types of economic and demographic factors 
that affect these outcomes, and the data that measure these elements. 

3- We de?vcribe a research design for analyzing the net impact of the 
JTPA. The research design describes how to select the most reliable 
comparison group of otherwise similar nonparticipants to approxi- 
mate what the labor market experiences of participants would have 
been in the postprogram period had they not participated in the 
program. This approach to net impact analysis attributes to program 
participation the im rcmvntul gain in labor market experiences that 
occurs over and alwve what would have happened had these indi- 
viduals not participated in the program. This is the appropriate 
concept for providing infonnation on the return on investment of job 
training programs. The rcsearch design also specifies how to select 
a representative sample of JTPA participants as well as sufficient 
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numbcrn of participants and comj^siHi group members to provide 
statistically ^Mid net impact results. 

4. We discuss the steps involved in obtaining and processing tfie 
required data. In particular, ^ describe the elements of a data col- 
lection and prwessing cycle, and indicate potential problems that 
may arise. We then describe how to organize the various daui 
sources, the types of data cleaning to be perfomicd. and key featui^s 
of the analysis files to be constructed. 

5. We outline a data analysis plan for estimating the net impacts of 
JTPA programs on participants' postprogram outcomes. Beginning 
with a description of methods to determine the quality of any com- 
parison group selected, we iteMrribc an approach to estimate the 
overall net impacts of ITPA and major tai^get groups with respect to 
the key participant postprogram outcomes indicated in the federal 
legislation. We also discuss potential threats to the validity of the 
analysis and indicate pi>ssible approaches for adjusting for such 
problems. 

6. We describe how to conduct a cost analysis to determine the costs of 
JTPA services. Wc also indicate how to combine this information 
with the net impawi a*suUs to provide evidence on the return on 
investment of job training projects. 

7. We indicate how the net impact analysis complements analysis 
strategies described in otl^r chapters. 

8» We conclude with a summary of the general concepts and methods 
applicable to a net impact evaluation of any social program. 

Learning fhim Past Evaltiations: I^veloping 
the Context for JTPA Net Impact Analysis 

In developing a strategy to evaluate the net impact of any social 
program, it is very important to understand the various approaches that 
have been used previously and the nature of the results that have been 
obtained. In an earlier net impact evaluation guide (Johnson 1986), a 
detailed review of approximately 20 employment and training net impact 
evaluations was provided. Tliis section summarizes the lesuhs of that 
literature review to set the context for the conceptual framework de- 
scribed in the following section. 
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With ihc excepiion of several recent net impact evaluations of CETA, 
most of these studies examine the mps^t of CETA's predecessors, for 
example, the Manpower Development and Training Act (MDTA) class- 
room and on-the-job training programs. Some evaluations coiKcm 
CETA contemporaries, such as the Work Irw^ntive (WIN) program, the 
Job Corps, or various employment and training demonstrations. Though 
nofK^ of these studies deals explicitly with JTPA programs, and many 
focus on pre-CETA programs, they are of interest because the programs 
examiiml have many characteristics in common with JTPA pmgrams, 
and the evalurtion elements us4^ arc the same as those to be developed 
for a JTPA nwt im^^ct model. 

Hie previous net impact evaluations of employment and training 
programs examine the impacts of services on participants* earnings 
almost exclusively, which is consistent with the policy objectives of 
federal employment and training legislation/ Moreover, Social Security 
Administration (SSA) records have been the main source of earnings 
data. Although SSA earnings records have several advantages (e.g., they 
are a cost-effective source of longitudinal data measured comparably for 
participants and comparison group members), they have several poten- 
tial disadvantages, including coverage problems, exclusion of earnings 
beyond the taxable maximum, and delays of up to three or four years in 
obtaining reliable data. These delays prohibit a state-level evaluation 
from obtaining policy-relevant rejoilts in a timely fashion. In addition, 
when SSA earnings are the only outcome measures available, evaluation 
is limited to estimating imj^Kicts on m annual basis.^ 

The results of tln^se studies generally indicate large net earnings gains 
for women, particularly nonblack women; whereas, the net effect of 
employment and training programs on the earnings of adult men is less 
clear. Although almost all studies find the net earnings gains of men to 
be considerably less than those obtained by women, several recent evalu- 
ations fmd that male trainees never regain the earnings fx>sition they held 
prior to training relative to otherwise comparable nonparticipants. If this 
is true, why did men continue to enroll in these employment and training 
programs? Perhaps because of program earnings and the substantial 
training subsidies offered by CETA. 
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Thej^ nesulis indicale ihai scparaie net impaci models should be 
developed for men and u omen because Ihe aMaiionship between earn- 
ings and various scKioccononiic and demographic characierisiics clearly 
is different for men and women. 

Although the earlier evaluations of MDTA programs focused more on 
5«paraie net impaci models for whites and blacks, more recent evalu- 
ations of CETA programs focus on estimating separate models by age 
groups. Because of the difficulties in developing valid net impact 
estimaies for youth, many recent studies have estimated models only for 
adult men and women. The problems for youth are twofold: earnings are 
not the single appropriate outa>me measure for youth, as the relative mix 
of schwiing, market work, nonmaiicet work, and leisure evolves rapidly 
over time for them: and it is very difficult to draw a aMiable comparison 
group for youth with limited and highly variable earnings histories. As 
such, ii may not be feasible to develop a stjte-level JTPA net impact 
mcxlel for youth that provides valid results. 

Another evaluation element concents the participant groups chosen 
for analysis and the variables includc*d in the model to measuiv the 
service intervention or treatment effects. For the most part, these studies 
focus on estimating the average impaci of program services on earnings 
for the selected subgroups. Because in many cases subgroups of interest 
were j^rticipants enmlled in specific program activities (services), this 
resulted in numea)us net impact estimates by program activity or 
ireaimeni. The only other dimension of the treatment examined in some 
of these studies was length of pn>gram participation. TTie results indicate 
that net impacts vary by program activity and length of participation. 
Although fewer programs are generally offered under the JTPA (as 
compiircd with CETA), and the average length of slay in JTPA is less 
than in CETA. it is still imjxmant to develop models that examine these 
potential differences in outcomes. 

Finally, because virtually all of these studies rely on large-scale data 
bases on program jnirticipanis develop^ well before the analysis was 
undertaken, liiile information wax generally available on the content of 
the services provided, whether the program services were pmvided as 
planned, and the extent to which the services pn)vided varied across sites 
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and over time. To overcome this problem, it is important to conduct a 
(HDcess anidysis like the one (fescnbed in chapter 4. 

Developing a Conceptual Framework 
for Evaluating Net Impwrt 

The conceptual framework is tte fim major element in the design of 
a net impact evaluation. This framework answers the question '"What is 
to be learned frcmi the evaluation?'' and identifies key research questions 
to be addressed. These arc: 

1 . Outcomes to be e^^amii^ and the relationship of these outcomes to 

2. program activities or services. 

Participant groups and program activities to be incluckd. 

3. Specific defmitions of the variables that measure outcomes, program 
4- activities, and any other variables affecting the relationship amcmg 

activities and outcomes. 
As such, the conceptual framewoiic guides the development of the re- 
search design ar^ analysis methcKl. This section describes a conceptual 
framework for conducting a state-level JTPA net impact analysis that 
accounts for the lessons learned from previous studies and is based 
entirely on available stale administrali/e data sources.^ 

Ctwral Evaluation Questions 

Although employment and training programs funded under the JTPA 
can affect different groups, such as participants, employers, the govern- 
ment, or society as a whole, in various ways, the primary goal of tl^ state- 
level JTPA wt impact model is to determine tte extent to which JTPA 
program activities or services improve the labor market experiences of 
participants relative to what their experiences would have been in the 
absemre of the program. TIk net imf^t of JTPA program activities on 
participants* pK>stprogram labor market experiences provides poli- 
cymakers with an indication of the overall gains due to ttese [m:>grams« 

Although it is important to know whether tte mix of JTPA activities 
on average is effective, for policy purposes it is equally important to 
identify the relative effects on different target groups. For example, to 
improve targeting it is important to know whether certain participant 
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groups benefit more from a particular JTPA activiiy or service than otter 
groups, and whether the net impact of JTPA differs among program 
activities, by length of participation, and by local economic conditions. 

Tlte general objectives of the state-level i^t impm model can be 
summarized by (he following key research qire«tions: 

1 . What is t}» overall ret impact of JTPA program activities (services) 
on participants' postprogram labor market experiences? 

2. Do the net impacts change over time? If so, in what way? 

3. Which program activities result in the largest net benefits to partici- 
pants and society relative to their costs? 

4. Which groups gain most from participating in JTPA? 

5. For a given program activity, do individuals who remain in JTPA 
longer experience greater net gains in labor market outcomes? Docs 
this effect vary among activities? 

6. How does the net impact of the JTPA vary by local program and 
environmental conditions? 

Table 2. 1 details the key elements of the net impact conceptual frame- 
work that help to make these questions more sfxjcific. Each of these 
elements is discussed below. 

Measuring the Factors Involved 

in Answering the Evaluation Questions 

Participant Outames 

As table 2. 1 shows, the state-level JTPA net impact model focuses on 
specific postprogram participant outcome measures related to earnings 
and welfare dependency that are available in automated administrative 
records in most stales. Because unemployment insurance (Ul) Wage 
Records are maintained on a calendar quarter ba.sis, the earnings meas- 
ures for longer periods can be created as the sum of quarteriy earnings 
amounts. The Aid for Families with Dependent Children { AFDC) grants 
measures can he calculated as the sum of monthly grant payments. The 
employment status and the AFDC participation status measures are 
defined as "dummy" (or indicator) variables. For example, a persc«i is 
defined to be employed for any period of interest if UI Wage Records 
indicate positive earnings were received during the f^riod ( I if em- 
ployed, 0 otherwi.se). 
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X^r E^itirti of tte (^i^ti^ FnKimmk 
for JTM Net impiti Mdtfil 

Dotone Momros 

• Qviilrty. semiannual ami annual earnings and employment statifS b^sed on Ui^mpioyment 
li^rance (Uf ) V^e Rec^mls 

• Quarterty, scmimnmial and annual AFDC gt^nts and AFOC partrci^^tion status based on public 
is^stance (PA) records 

• Adult men and women: to tf% extent {H)ssibie. dt^ggregate net tmp^ts by ^x, age. race d 
ettmfcity, education, and for women, by welfare status. 

Pn^fMi Actives Iff SifficiS 

• C^r(K)m tmintng: wtten prt^ram data are available, ^imate ^parately for remediat education 
ami basic skHls versus ^lecif rc occupational^skHis trammg, 

• On-the-job training. 

• Job search assistance. 

• All other acttvit!es or services. 

• Con^binations of tt^ above actfvitres or services 
Ltki^ Msrtet ComtWons 

• Unemployment rate. 

• Urban or rural location 

• JTPA Management information System (MIS) participant charactenstics. prc^ram actwrties, and 
placement experrences. 

• PA Grants Records: wtiether recerwd AFOC and size of AFOC grant 

^ UIWageR^rds: whether employed and amount ot giarteriy earnings 

• Ul Bej^fit Hrstory: whether received Ul benefits in preprngram penod. 

• Local labor market infofmation: the IwaMor regional) unemployment rate. 



These outcome measures are consistent with the major objectives 
the JTPA legislation as specified in the law. As the literature revievv 
indicates, these measures typically extx*ed the natua* and extent 
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outcomes examined in previous studies of the net impacts of subsidized 
employment and training programs. They alsocapture the range of short- 
term and relatively long-term impacts that could be observed wiihin, ap- 
proximately, a two-year program analysis cycle. 

The administrative d^ sources for ccmstructing key participant out- 
come measures have several advantages for evaluating the net impact of 
social programs for which earnings aiKi welfare (tepentto^y are of 
interest. First, VI Wage Records are not truncated at some taxable 
maximum; that is, actual e^nings are reported, which reduces n^asure- 
ment enxM-. Second, UI Wage Records are available by calemlar quarter, 
with only a three- to six-month delay in obtaining reasonably complete 
data. The availability of data by calendar quarter altows considerable 
flexibility in the choice of a posiprograra follow-up period. Third, UI 
Wage Records are not subject to interviewer bias or respondent error. 
Moreover, these data are not subject to problems arising from siMne 
respondent reporting net (after-tax) earnings and others reporting gnoss 
(before-tax) earnings. Finally, they are not affected by response-rate 
IWDblems. 

Monthly AFDC grants from administrative records have several ad- 
vantages relative to data olHained through surveys. Hiese include 
timeliness of data availability and the absence of respondent reporting 
errors, interviewer biases, arel respcmse-rate problems. Becau^ these 
administrative data systems arc relatively inexpensive to obtain, our net 
impact analysis relies cm them.^ 

It is also important to n^ogxiize the limitations of these administrative 
data sources. First, although the Deficit Reduction Act of 1 984 required 
all states to become wage-reporting by 1988. there are still a few stales 
that do not report at this time, and will not for the most part be able to use 
the net impact model to examine earnings impacts." Second, stales that 
do not have automated AFDC Grants Records available at the slate level 
cannot easily use tlK net impact model to examii« JTPA imi»cts on 
reducing welfare dependency. TTiird. because our net imp^ appro^h 
relies Ittavily on the availability of historical earnings and grants data, 
states must be able to directly access or retrieve from archives two to 
three years of UI Wage Records and public assistance (PA) Grants 
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Records for a given person at any one time. Although most UI Wage 
Records systems arc similar ^rross wage-reporting states ai^ generally 
comain at least seven quarters of data at any time, there is considerable 
variation in state utd local welfare administration md record-keeping 
prwrtices, as well as differences in the degree of data automation and 
retrieval capabilities. These variatitms present obstacles to implement- 
ing this compcMient of the net imp£u:t analysis in certain states. 

Fourth, UI Wage Records do not generally include employees of 
federal, state, or local governments. %if-employed individuals, or em- 
ployees in certain other occupations.' Finally, because the system is 
state-ba$«i, it is impossible to distinguish individuals who work across 
the bonter in a different state frwn individuals who do not work in 
covered employment. Thus, one should be very careful in estimating the 
net impact of the JTPA on earnings for large service delivery areas 
located near state borders. 

Particifmnt Suhgnntps 

The next conceptual ii^sue concerns the participant groups of interest. 
As table 2. 1 shows, the state-level net impact analysis is limits to adults 
because earnings, employment, and AFDC dependency are not the ap- 
propriate cH'»come measures for youths or tht^e entering ibe labor market 
for the first time in their life cycle. Moreover, the more appropriate of 
these measures, such as schooling attainment and employment compe- 
tencies, are not contained in any existing program data sets for both 
program participants and comparison group members. In addition, as 
discussed above, there is extensive evidence indicating the difficulty in 
(teveloping a reliable matched comparison group for youths (Dickinson, 
Johnson, and West 1987(b); Lalonde and Maynard 1987). Because of 
these problems, the net impact analysis is (tesigi^d only for adults. States 
interested in examining youth issues .Uiould con.sidcr other approaches 
for conducting a net impact analysi.s.'" 

A second issue concerns whether separate net impact analyses need be 
developed for any specific adult groups. Sample sizes and state re.sources 
permitting, separate net impact models should be developed for adult 
men and women because the relationship between earnings and other 



ERIC 



58 Evaluming Sixriui Probicmi^ 

dcmc^phic characteristics is very different for ihesc two grmips. In 
Titian, it is desirable to es^timate se;»raie net impact models by race 
and other panici{^t groups, sample size permitting. If no statistical 
differences arc found ^ong any set of population graups, the gnmps can 
then be combined for subsequent analysis. 

Rnally, it is important to investigate whether the impact of JTPA 
varies by the following particijiKint characteristics: 

1 . Age— less than or equal to a particular age (e.g.. age 35 ) as compared 
to over age 35. 

2. Ethnicity^ — whites as compared to blacks and Hispanics. 

3. Educational level — at least high school graduate as comjwt^ to 
nongraduates. 

4. Welfare status for adult women — welfare recipients as compared to 
nonrecipients. 

The techniques for conducting such analv sis are described later in this 
chapter. 

Prof^ram Activities: The Senile Interventions 

The next important element of the conceptual framework is tlw deter- 
mination of the key treatments — program activities or services — to be 
assessed^ and the development of consistent defmitions of these variables. 
Section 204 of the Job Training PartMrship Act sets forth a lai^e array 
of potential activities. However, the ma^r activities provided umier 
jTPA are classroom training (CT), on-the-job training (OJT), araJ job 
search assistance (JSA). Nearly 90 percent of aduh FY 1984 Title II-A 
enmliees participated in at least one of these program activities. Al- 
though it is important to include in the analysis those participants 
assigned to all types of activities in order to assess the full range of JTPA 
activities, as table 2.1 shows, it is most useful to examine the separate 
elfects of Cr, OJT, and JSAJ^ A brief description of each of these 
program activities is provided below. 

Ciassroim! training involves bask or remedial education or occupa- 
tional-skills training to ensure that individuals acquire the ability and 
knowledge necessary to perfomi a specific job for which there is a 
demand. Such programs are usually provided in a classroom or an 
institutional setting off the job. 
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On-the-job traimn^ emphasizes the development of occupational 
skiliK in on actual work netting, ideally in the private f^ector. The 
programs are designed for participants who have been first hired by the 
employer, and the training (xrcurs while the participant is engagol in 
productive work that provides knowledge or skills essential to the 
adequate performance of the job. 

J()h search assistance includes any activity that focuses on tiK! devel- 
opment or enhancement of employment-seeking skills. This service is 
provided to participants who need practical experience in identifying and 
initiating contact and interviewing with pn>spective employers. It is 
usually conducted in a structunxl setting and can include approaches 
such as job-finding clubs or instruction for self-direct<^ job search. 

Several complications arise in developing measures of independent 
variables to represent these progmm activities. First, there is consider- 
able variation among SDAs in the chai^'teristics of specific program 
activities, such as length of assignment, occupational category of the 
training, hours of training per day, and qualify of instructional inputs. 
This makes it difficult to sj^ify variables thai reprcswrnt a uniform 
treatment. When such a variable is not unifonn. it is difficult to interpret 
its statistical meaning, and biased estimates of net program impact can 
result. Nevertheless, one must aggregate genemlly similar activities 
because it is simply not possible to reliably estimate the net effects of tl^ 
virtually unlimited variations of a given program aciivity. 

Second, lai^e differences among SDAs in the nature of the prognim 
activities provided tltq likely to occur. For example, even within a state, 
work exfK^rience activities in a particular SDA may resemble OJT pnv 
grams in another SDA, Finally, the way in which the training activhies 
provided arc recorded in the MIS can cause complications. For example, 
due to the lack of uniform national rcjxirting rcquircmenis, some SDAs 
record participattcHi in a job search workshop as job search assistance, 
while others rccord it as classroom training because the sessions are 
conducted in a classroom setting. Such diflerences in the content and 
recording of program activities across SDAs emphasize the importance 
of conducting a process analysis concurrently with the t^t impact 
analysis in order to develop meaningful and consistent measures of 
program activities. 



ERIC 



60 Evaluating Social Fhx>b]em.s 



As indicated above, the ways in which the treatment variables ai^ 
defined are in large part determii^ by tte structure and content of the 
SDA Management Information System (MIS) and how standardized 
definitions are acn^s SDAs. In i^dition, any variable definition cbpends 
on the specific research questions of interest and the population size of 
individuals who fimrticipate in a given program activity. For example, it 
is desirable to separate claHsrxK)m training activities that focus on 
remedial education and basic skills from classroom training activities 
that provide specific occupational-skills training. However, if the 
number of individuals participating in each of these programs is too small 
to produce statistically reliable net impact estimates for separate activi- 
ties, it may be necessary to collapse these two treatment variables into 
one that represents cls^sroom training pr(^ram activities in genei^. 

Thus, although the specific definitions of the treatment or piDgram 
activity variables depend on several factors, the following variables 
should be specified to examine overall impacts and impacts by program 
activity and other characteristics of the services provided: 

1 . Participant dummy variable: 1 if JTPA j^icipant, 0 otherwise. 

2. Classroom training dummy variable: 1 if CT particij^t, 0 otherwise: 

~ remedial ^ucation and basic skills dummy variable: 1 if CT 
program in remedial education or basic skills, 0 otherwise; 
— specific occupational-skills training dummy variable: I ifCT 
program in a specific occupational skill, 0 otherwise* 
3- On-lhe-job training dummy variable: 1 if OJT particii^t. 0 crtherwise. 

4. Job search assistance dummy variable that includes all employment 
and placement-related activities: 1 if JSA iwticipant, 0 otherwise- 

5. Other ^tivity dummy variable: 1 ifnotaCT, OJT, or JSA participant, 
0 otherwise. 

6. Multiple-treatment dummy variable: 1 if a combination of two or 
more of CT, OJT, and JSA, 0 otherwise- 

7. Occupation of training dummy variables: 1 if in specific 1 -digit Die- 
tiotiary of Occupational Titles (IX)T) code, 0 otherwise, for each 
type of occupational skill, 

8. Length of program participation in weeks. 

9. Total training hours, the pixxiuct of length of program j^rticipation 
in weeks, and the n'^mber of training hours per week. 
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The Program's Environment 

'Hjc final conceptual element concerns the program envimnmenlal 
conditions that should be included in the net impact analysis. Here, we 
refer primarily to chai^cterisiics of the labtir markei(s) within which the 
pn>gram operates, although major SDA characteristics can also be 
considered. 

Little is known about how the net impact oi employment and training 
programs varies by the environmental conditions surrounding a program 
at a point in lime or over lime. In addition, because of the nature of local 
program envimnmenial condition.s (i.e.. there may be no within-SDA 
variation on these conditions), it is important to recognize that, at best, 
it will only be possible to obtain reasonably precise estimates of a few key 
conditions. However, this can be done only in states that have a large 
number of SDAs and exhibit considerable variation in the conditions 
among SDAs. 

In conducting a net impiict evaluation of JTPA programs, it is most 
important to control for differences in local unemployment rates and 
Unation, i.e.. whether the program participants and their comparison 
group members arc located in an urfnjn or rural area. These factors are 
likely to affect the key outcome measures: the employment and earnings 
of adult men and women. 

The unemployment rate can be obtained from the Local Area Unem- 
ployment Statistics, published by the United Slates Bureau of Labor 
Stati.stics (BLS). This information is available monthly at the slate and 
county level and for over 1 ,0(K)cities with a pi>pulation of ai least 2.1.000. 
Aggregate measures of the unemployment rate corresfwnding to the 
quarterly, .semiannual, and annual outcome periods of interest can be 
calculated as an average of the seasonally adjusted rates. 

In specifying the unemployment rate variable for an SDA, it is also 
important to recognize that monthly values will not generally )x avail- 
able for the preci.sc area of interest. Expending on the geographic 
jurisdiction of the SDA. the area it .serves may be larger or smaller than 
the county or the city for which any given information is available. 
Where the SDA serves multiple ctjunties. one should calculate the 
appropriate labor market variables by aggregating over the counties 
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served by the SDA, Forcxample, tocalculatc ihc unemploynicnl mteone 
would simply sum the number of individuals unemployed in the various 
counties served by the SDA and divide by the total number of individuals 
in the labor force in those counties. In ca^^s where the SDA serves only 
part of a given county, and where no value is available for a smaller 
geographical area such as a city, one is constrained to use the county 
value. 

It may also be possible to provide some infonnation on the manner in 
which the net impact of JTPA varies by different managerial, organiza- 
tional, or SDA strategies. The service delivery strategics examined 
should be based on their policy importance to the particular state doing 
the analysis. Moreover, to ensure that the strategies of interest are 
distinct and quantifiable, and that there is sufficient variation among 
SDAs to support the analysis, it is imiK>itant to conduct a process 
analysis. Thus, if states with a large number of SDAs (roughly 30 or 
more) arc interested in obtaining information on how the net impact of 
JTPA varies by a key service delivery strategy, they should first ensure 
that significant differences in this strategy exist among SDAs. 

If it is possible to quantify these differences (usually by means of 
dummy variables), one could then use the measures of these variables to 
determine how the net impact of JTPA varies among SDAs. In states 
with relatively few SDAs, it is unlikely that such an analysis would 
provide sufficiently precise estimates of the differential effects of the 
strategy of interest to warrant the analysis. 

Developing and Implementing a Research 
Design for Evaluating Net Impact 

To provide valid estimates of the net impacts of JTPA programs on the 
earnings and AFDC dependency of adult men and women, ihe research 
design must contain several elements. Table 2.2 provides an overview 
of the key aspects of the research design that are discussed in detail in the 
next several sections. Although the specific details of this research 
design are sensitive to the features of JTPA, its basic elements and tfMJ 
issues to be considered in making decisions ate applicable to any net 
imi>act evaluation of a social program. 
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nBmr^ O^ft for tt» JTPA Itet imim! MMal 

• Samples of JTW participants {axiuH mn and women) must chosen so that the results can tm 
gen^atized to the si^e Imrei. 

• Valid comparison groups must be chosen so that the impact of JTPA can be drstinguished from the 
impacts of other factors tt^ afso affect earnings and wetfare def^rrdency . 

• The size of the panicipanl and comparison samples must t>e determinoJ so that program impacts 
can be n^ured wiOi pr^mion. 

Mb Wiiett}i9 

• Comparably measured pneprogram and postprogram (tata for partrcipams and comparison group 
members must be obtained from several different sources, processed, and analysis fites developed. 

• The comparison groups must be examin«J in detail to determine their comparability to the 
parterpant groups and to identify any adjustments that may nawJ to be made to correct for selection 
bias. 

• A comprehensive strategy must be de^^lof^ to provide valid estin^tes of the net impacts of JTPA 
activities {^rvices) on tt^e po^rogram outcomes of parhc^nts. 



The elements of the sample dej^ign are discussed below. Data 
collection and data pitxressing issues arc the subjects of the following 
section. A description of the overall estinmtion strategy and the specific 
net impact models to be estimated is provided in tlK! subsequent section. 

Sample Design 

The sample design is a key clement of the ovct^ll research design. The 
sample design involves the selection of the participant samples, a 
strategy for developing tt^ comparison groups, and the determination of 
sample size -lequiremenis for the analysis. As such, the sample design 
directly affects il^ internal and external validity of the analysis, as well 
as its statistical efficiency. Table 2.3 summarizes the three major 
elements in the sample design for a JTPA iwrt imp^t analysis model. In 
the text we discuss each of these elements in more detail 

Participant Groups 

TT>e major issues in selecting the participant group concern ( 1 ) the 
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tndividualH to be included in the sample frame. (2) }K)tential sample 
exclusions, and (3) the prot^edure to select participants from the sample 
fnune for imrlusion in the analysis. We discuss these issues below. 

Sample frame. The choice of iIk sample frame is an important 
determinant of the degree to which the findings can be generalized. The 
sample frame should represent all JTPA participants so t!^ analysis 
results can be genemlized to the state level. The net impact model 
includes in the participant sample frame all adult male and female JTPA 
enmllccs during a given timo intervaL as indicated in table 2.3. In 



Table 2.3 

Key Eldmanis of tts Sampit Dtsign tor 
the JTPA itet Impact fltodii 



PaiUfiipafl! Groap 

• Comprised of samples of adult men and women who moW in JTPA in each quarter of a given 
program year. 

• IndividuaJs will be exclude from the sample frame it they are not from 22 to 64 years ot age. 
Individuals will sub^u^ntty from the analysis sami^s if they have mssing data on 
it^ JTPA servm rw^ived (e.g.. program activity. length of participation). 

• Ouartc. iy samptes of JTPA participants will be selected randomly from the groups of adult men and 
women etirollees that are included in the sample frame to ensure that tte sample is representative 
ot JTPA participants in the state. 

(k>m]»rts<m fifosp 

• Compns^ of samptes of adult men and women wtio are new ES r^tstrants m offices m the areas 
served the SDAs in each (luarter of a given pn^ram year. 

• Individuais will e«:luded from the compans(W sample frame if they are not from 22 to 64 years 
of age. if they are not ectH^onricalty disadvantaged, or if th^ particfpate in JTPA. 

• Ouarterfy samples of compan^wi group members will selected from ttie sample frame of new 
ES registrants using a Gratified random pn»^ to ensure that ES r^istrants and JTPA {^rtidpants 
are similar on certain Key characteristics (e.g.. welfare recipiency, U! rrciplency}. 

SanqMoSiza 

• Becau^ the additional cost of inert ^^mpies^^mverysmali.stat^areencouragedtoinclude 
in the analysis as many t^rtlcipants .ompartson group members as their staffs and computer 
r^urces can handte. 

• As a guideline, a total analysis sample of 12.CXH} cases — divided equally betv^en adult men and 
women, and t»tw«) participants and comparison group members {i.e. 3.(K)0 each)— should 
adequate to most state's analy^s needs. 
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particular, aduli men and women who enroll in JTPA in each calciular 
quarter of a given program year are to be sampled separately. 

This frame has several advantages. First, it yields a representative 
sample of JTPA participants in which neither short-term nor long-term 
participants are oversampled. one which is not sk^nsitivc to seasonal 
differences in the characteristics of participants or program activities. 
Second, because the time period for selecting each participant cohort 
within the program year is not too long (e,g,, tha^c months), it should be 
IX)ssible to select quarterly samples of c%)mparison group members that 
closely match participants on the timing of the preprogram decHt^ in 
earnings. This is particularly important for ensuring valid net impact 
results. Third, using an enrollec-hased sample maximizes the amount of 
preprogram earnings and AFDC data available for the model. Fourth, 
this iq>pn>ach accounts for the fact that males and females have qualita- 
tively diffenent labor market experiences 

A participant group comprised of aduU men and women who enroll in 
JTPA in each of the four quarters of a given program year has implica- 
tions for the timing of project results and the length of the postprogram 
observation period within an approximate two-year program analysis 
cycle. With such a sample, one can obtain net impact estimates for the 
period one year following the calendar quarter after termination only for 
the first quarter cohort, and only a three-month net impact estimate can 
be obtained for all four quarterly cohorts in approximately a two- to two- 
and-one-half^year analysis cycle. Of course, by obtaining additional 
postprogram outcome records for sample members, one could estimate 
longer-term impacts by extend : analysis period. 

Sample cMiusions. Once thi le frame is chosen, one must then 
determine whether certain typt .ndividuals should be excluded. 
Although such exclusions reduce th^ representativeness of the partici- 
pant sample, exceptions may be indicated for several reasons. It may be 
desirable to exclude cases that lack data on critical items, or the 
availability of extremely small samples of atypical treatment or partici- 
pant groups may engender exclusion. 

Most studies incoriH>raie resections on participant age. Although 
there is no universal agreement on the age range to use, participants under 
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age 16 have been excluded because earnings are not an appropriate 
outcome measure for individuals who are likely to return to school. 
Participants age 6S and older have been excluded becau% participatiim 
in employment ami training programs among individuals eligible for 
retirement is rare, and a valid comparison group is hard to identify. 

Because the net impact model focuses m adults only, we restrict the 
participmnt sample to individuals of at least age 22. Because it is difficult 
to obtain a valid comp^sK>n group for older [^rticipants, all individuals 
age 65 and older should be excluded frcmi the {^icipant sample 1 rame. 
In addition, if the JTPA pn^rams in the state ^rve very few individuals 
over age 55, it is desirable to exclude individuals over this age. 

Individuals should also be excluded from tl^ sample frame if they 
have missing data on key variables.^"^ A more difHcult problem arises 
when information is missing on the ircatment provided by JTPA, For 
example, one cannot estimate the net impact by program activity or by 
length of stay for individuals with missing information on program 
activity or for those who have incomplete data on the start and end dates 
of JTPA participation, A few problems concemmg the omission of 
program start and end dates may arise, in part, because lengln of stay 
information is necessary for adjusting certain perfonnance stamJards for 
JTPA Title II-A programs. However, since there are no reporting 
requirements concerning program activity, some cases will cc«itain 
missing or unusable program activity information. 

Moreover, because the quarterly enrollee samples will be selected on 
an ongoing basis, one cannot know for sure how many cases must be 
excluded for missing data problems until after the JTPA MIS data arc 
obtaitied for the sample selected. To compensate for the resulting sariple 
reduction, an ex^^nded p^icipant sample should be selected initially* 
Individuals who are subs^uentiy detemiined to have missing data on 
key JTPA services must be excluded from the analysis sample, provided 
the reason tl^ items are missing is not systematically related to the impact 
of tte program.^** 

A final issue concerns whether to exclude individuals who j^iiicipate 
in JTPA for only a minimal peri(^, $\^h as less than a week. Some 
studies have included all employment and training participants in the 
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analysis sample while others have impt>sed arbitrary restrictions that 
exclude individuals who participate in the program for some minimal 
period* Note, hiwever, that more-motivated individuals leaving JTPA 
early because they have found jobs, excluding them from the analysis 
would result in a negative bias in assessing JTPA impacts. This is 
because those participants who would do relatively well on Uieir own 
would he excluded from the i^icijxmt sample, while tin: same types of 
individuals would still be included in the comparison group smnple. It 
is also |K)ssible that short-term particif^ts might cM:)nsisi of individuals 
who would do less well on their own thaii other JTPA participants, 
Because similar individuals would remain in ite comparison group, 
excluding the less-advantaged individuals from the participant group 
would result in a relatively more-advantaged participant group and a 
positive bias in the estimated program impact. 

Either scenario yields a selection bias that threatens the internal 
validity of the analysis. As such, we recommend that ihe sample of JTPA 
participants be kept as representative as possible and that cases not be 
excluded based on length of stay in the program. It is then possible to 
examine whi*ther, and in what ways* short-term participants differ from 
long-term participants to better understand the nature of any selection 
bias. This, in turn, will help to detciroine the degree of confidence to 
attach to net imp,*ct estimaies by length of program participation. 

Selec ting the partU ipum sample. Once the exclusions from the par- 
ticipant sample frame have been detemiined, the next step involves 
selecting JTPA enrollees for inclusion in tte analysis sample. In some 
states all enrollees in a given program year will be necessary to provide 
reasonably precise estimaies of the average effect of JTPA programs. 
The sampling issue primarily arises in slates that serve large numbers of 
JTPA participants. We outline a method below for selecting a participant 
sample from the sample frame described above. 

As indicated in table 2.3, the quarteriy samples of JTPA participants 
slKHild be randomly selected from groups of adiilt men and women 
enrollees in the sample frame- However, the proportion of males and 
females varies ccmsiderabiy across SDAs. Therefore^ one must first 
stratify the participant sample by gender before the analysis samples are 
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elected, iMhcrwisc there may be insufficient numbers of either men or 
women for analysis purposes.'"' Chcwsing random samples in this 
manner also has the major advantage of providing separate representa- 
tive samples of adult men and women partieipants. so that the results ean 
be generalized by gender. Moreover, estimates of !he net impacts of 
JTPA by gender and of the differential im^^is b\ program aetivity 
separated by gender can be obtained without weighting the sample. Also, 
by selecting participants randomly, an analysis of progt^m activities 
assigned to different types of individuals is possible. 

In atkiition to stratifying the sample by gender, slates that want to focus 
on specific groups, such as female welfare recipients or male high schwA 
dropouts, may also consider sti^iifying the participant sample and over- 
sampling the groups of interest. In general, stratification is desirable only 
when the research questions of interest relate to gn^ups that txrcar rarely, 
or that occur so frequently that their nonoccurrence is rare. Depending 
on the specific research questions, one could stratify ;>n the basis of par- 
ticipant characteristics or by program activities i .rvices). 

For example, because of the wide variation across states and SDAs in 
the u.sc of work experience programs, stales intemsted in examining the 
net impact of these programs would probably need to stratify and over- 
sample participants. Moreover, because job search assistance gemrrally 
constitutes a less intensive treatment and is therefore likely to have a 
smaller net impact, a much larger sample of participants in JS A is needed 
to precisely measure the smaller expected effect. Thus, states interested 
in precisely measuring benefits gained fmm JSA participation, must 
sample program participants in greater numbers. States interested in 
stratifying the participant sample and oversampling certain groups 
should consult a sampling expert to understand the steps in drawing the 
information and the implications for conducting the analysis. 

Crmipijrison Groups 

To estimate the net impact of JTPA on participants* jx>stprDgram 
outcomes, a method is needed to gauge the rc.sults participants would 
have experienced had they not participated in JTPA. The ideal research 
design for measuring the net impact of any social program involves the 
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use of a classical experimental design to develop a true control group. In 
such a design. JTPA eligibles wouW be randomly assigned lo cither a 
tneatmem group that could receive JTPA services, or a control group that 
could no*. With this method, the only systematic difference txrlween the 
two groups is receipt of program services: any differences in program 
outcomes are due to JTPA. However, ethical and legal concerns can 
preclude the use of a randomly assigned control group. Thus, we develop 
an alternative method that approximates a true control group to the 
maximum extent possible. 

A standard appn>ach for delemiining the net impact of a program is to 
compare experiences of persons treated by the program (i.e.. JTPA 
participants) with experiences of otherwise? similar persons who arc not 
treated by the program {the comparison group). The comparison group 
is used to estimate what the experiences of the participants would have 
b^n in the postprogram period had they not i^icipaied in the program. 
To ensure that differences between the experiences of the two groups can 
be attributed to the program, the comparison group must have character- 
istics similar to participants, particularly in terms of program eligibility. 
Moreover, available data must be comparably measured for the two 
groups. One must also \ crify that individuals in the comparison sample 
in fact did not receive JTPA services. 

As shown in table 2.3, the comparisiMi group should be comprised of 
new Employment Service (ES) registrants in offices in the arcas served 
by the SDAs. ES registrants have several advantages as a comparison 
group. First, data are available on several characteristics of interest — 
including those related lo JTPA eligibility — that generally are compara- 
bly measured with JTPA MIS dala.^' Second, like JTPA participants, 
IKJW ES registrants have probably experienced a recent decline in 
earnings. 

Fmally. also like JTPA participants, ES registrants are in il^ labor 
force at the time they apply for assistance. That is, they are either 
working or actively seeking work. It is important to ensure that 
pwticipant and comparison group member* arc similar in their attach- 
ment to the labor force. Otherwise, net impact estimates can be 
errowous. To ensure compambility on preprogram labor force involve- 
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ment. the comparison group nhould be drawn fnmi new ES registrants in 
the same calendar quarter that participants enroll in the JTPA. 

Although ES registrants have several important advantages as com- 
pariHon group sources, certain disadvantages exist. First, because of 
recent reductions in federal leporiing requirements related to the ES, 
states are no longer required to submit infomiation on the number of 
economically disadvantaged applicants registered and served by the ES. 
Because being economically disadvantaged is the major criterion for 
JTPA eligibility, and given the importance of ensuring that tte compari- 
son group be similar to participants on all characteristics affecting 
eligibility, it is important that the economically disadvantaged status 
variable be available for the net impact modclJ** Fortunately, many slates 
apparently have continued to collect information on the economically 
disadvmitagcd status of ES registrants. States that no longer collect this 
information will have to modify the comparison group sample selection 
procedures, as ckscribed below. 

A second potential disadvantage to using ES rcgistrnjits as a compari- 
son group concerns limitations in procedures for relainiiii; historical data 
on these individuals. In the past, most states have kept automated data 
with individual ES records, including registrant characteristics and ES 
services received, for a period of three to five years. In some slates, 
however, individual-level daia arc purged after approximately one year, 
and archived backup tapes are not very accessible. In such states, it will 
be difTicult to draw the four quarterly samples retrospectively at one lime, 
as comparison group members for the first quarter cohort would already 
have been purged. Thus, such states must either draw the comparison 
samples on an ongoing quarteriy or semiannual basis, alter their purging 
practices, or retain historical data for 18 months to two years. 

A final complication encountered in using ES registrants concerns the 
possibility that the ES registrant file may be dominated by Ul claimants. 
In stales in which ES oHices are co-located with UI offices, or in which 
the policy is to actively monitor job search efforts of UI claimants, a large 
im^rtion of these claimants may be entered in tl^ ES registrant file. 
Because of the markedly different labor market experiences of the two 
groups, it is inappropriate to compare the oulcx>mes of JTPA participants 
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wilh a sample that is dominated by UI recipients. Thus, it is necessary to 
umtersample UI claimants in the ES registrant file in certain states to 
c^Hain a comparison sample with a projxjriion of UI claimants similar to 
the JTPA population. 

In adjusting for this problem, note that UI claimant status, as recorded 
on the MIS systems fcM- ES and JTPA, may not represent the same 
concepl. In particular, for JTPA participants, being a UI claimant 
typically means that the person has filed a UI claim and has been 
determined to be monetarily eligible. The ES claimant status refers 
simply to the filing of a claim for benefits and does not imply moi^tary 
eligibility. Because of this difference, a typical JTPA "claimant" is much 
more likely to receive UI benefits than a typical ES claimant. To ensure 
that UI rwripicncy is comparably measured for the two groups, the UI 
Benefit History file must be u^ to determine whether the person was 
a UI recipient. A decision on the appropriate rate for sampling UI 
recipifcnts from the ES registrant file would then be based on this 
measure. " 

Despite these potential disadvantages, we believe that ES registrants 
are the best comparison group source among existing state data bases. 
We now turn to a discussion of additional details related to drawing a 
sample of ES registrants. 

Comparison group sample exclusions. IMor to selecting comparison 
groups of adult men and women ES registrants, certain cases should he 
excluded from the sample frame to maintain comparability with partici- 
pant samples. In addition, it is desirable to exclude those individuals who 
arc clearly not eligible for JTPA and who are likely to have an eamings 
potential considerably different from JTPA participants. We discuss 
these sample exclusion considerations below. 

To maintain comparability with the JTPA participant samples, the 
group of ES registrants ^ouid be rcstncted to individuals over 21 and 
under 65 years of age. If it turns out that no one in the JTPA .sample is 
over a given age (for example, age 55), tlwn the ES registrant sample 
should be similarly restricted. Also. ES registrants must be excluded 
from the sample if ttey arc JTPA participants during eilter the prepro- 
gram, program, or pKstprogram period. TTiis problem, known as com- 
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parison group contamination, results in cumj^ring outcomes of program 
paniciiKmts with outcomes of other individuals who have participated in 
the program. Such a comparison yields biased net impact estimates, and 
results in understating the true imf^ts of the program. To minimize Uils 
problem, one should compare the Social Security Account (SSA) num- 
bers of current and rccent JTPA part«ci{^ts with the SSA numbers of ES 
registrants, and exclude ail matches from the comparison sample. 

A fmal issue concerns procedures to ensure the similarity of partici- 
pants and comj^son gnnip numbers on characteristics related to JTPA 
eligibility. As indicated above, the primary criterion for JTPA eligibility 
is that the person be economically disadvantaged. Over the last few years 
at least 95 percent of adults in Title II-A programs have met this criterioiL 
Moreover, of those who are not ecoiwmically di«^vantaged« or who 
were not certified to be economically disadvantaged* the act requires that 
they be persons facing demonstrated employment harriers. Because 
virtually all adult Title II-A enrollees are economically disadvantaged, 
the comparison group should also exclude all ES registrants who are not 
economically disadvantaged at application. This will help ensure that 
comparison groups arc similar to JTPA panicifKints on the key charac 
teristic related to JTPA eligibilityJ*' 

Setecting the l ompamon group samples. We recommend that the 
selection of .stratified nmdom samples of adult male and female ES 
registrants have the .same di.stributions a>i JTPA participants on certain 
key chanicteristics. This approach maintains maximum statislical power 
for the desired sample design, while ensuring that the participant and 
comparison samples are similar.^^ 

Because of program eligibility considerations and certain practical 
issues concerning the relationship between the ES, UI, and welfare 
programs, some of the more important characteristics on which to ensure 
participant and comparison group comparability are economically dis- 
advantaged status, receipt of UI bciKfils, and receipt of AFDC benefits. 
Because comparability between the two groups on ecoiK)mically disad- 
vantaged status will be ensured by excluding from the sample frame for 
the comparison group all new ES registrants who are not disadvantaged, 
no additional matching is required on this characteristic. 
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A simple random sample would probably include relatively Ick> few 
ES registrants »%'ho are AFEXT recipients, and relatively too many ES reg- 
istrants who are receiving Ui benefits. For example, nationally, only 9 
percent of adult men and 35 pea^enl of adult women JTPA lerminees in 
PY 84 were nrceiving AFIX" at application, and 15 percent of adult men 
and 8 percent of adult women JTPA terminees in PY 84 were UI 
claimants at application. On the other hand, it is probable that fewer than 
10 j^nrcnt of all ES registrants arc AFDC necipicnts and, in staicji in 
which the Job Service is co-located with UI, the fraction of ES registrants 
who are likely to be UI claim^ts could approach 50 percent. 

To ensure similarity on these important characteristics, comparison 
group members J*ouId be randomly selected from the sample frames of 
^ult men and women. Thus, for the separate samples of adult men and 
women, procedures would be used to make certain that the participant 
and comparison groups are similar on the proportions in the four cells 
representing combinations of AFDC and UI recipient status. Operation- 
ally, for a given total sample size of participants and comparison group 
members, sampling rates for each cell would be determined to match the 
two distributions, and then comparison group mcmbenj would be se- 
lected randomly from the cells M the given sampling rates as described 
in the next section. 

Sample Sizes for Partwipant and Comparison Samples 
An important element of the research design is the determination of the 
appropriate s^pie sizes for i^rticipant and comparison groups. As we 
indicated eariicr^ many states will have little choice concerning partici- 
pant sample size. Because the mai^ina) cost of increasing sample si/c is 
very low, even medium to large states should generally i»se the largest 
numbers of participants and comparison group membei^ feasible. In 
states with very large JTPA programs, howev-n"» samples should be 
drawn. This raises the issue of totp.l ?»mple size as well as aU<H:ation of 
total sample among the participant an -^oaristwi groups. 

The appropriate sample size for ' analysis ultimately 

depemis on the size of the imjmil tha^ et^t for fwlicy 

puiTK»es and the leve! of statistical accur<tc^ ,v , d. With larger 
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sample sizes, om has greater assurance of cictecting small differences in 
overal! outcomes between t)^ partici^mnt and comparison grouf^. as 
well as detecting differences for major panicipant subgroups or among 
program levities. The likelihocd of detecting a given difference in 
CHitccnnes also depots on the allocation of d^ total sample between tte 
two groups and the U!^xplained variamre of the outcome measure, such 
as earnings or AFDC grants. Thus, although the sample si;^ require- 
ments will differ for net impact evaluations that rely on other outcome 
measures, in general the mwie homogeneous the sample, that is, (he 
^nailer the variance of the outcome measure, the smaller the nimiber of 
cases i^ressary to detect a given difference in outcomes at a specified 
level of statistical significaiK^e. 

Based on a number of ctmsitterations, a total analysis sample of 
1 2,(XX), di vid«l ^ually between adult men and women, {wticipants and 
comparison group members (that is, 3,000 each) should meet most 
states* analytical needs*^^ This sample size recommendation refere to the 
final analysis samples and, because sonn? c^^s will be omitted for 
various problems described above, initial samples should t« somewhat 
larger. 

Stales interested in obtainmg more precise i^t impact estimates for 
subgroups of adult men or women, should consider larger sample sizes 
as nested. In addition, states with relatively smaU JTPA programs (i.e., 
fewer than 1,000 adult enrollees per year) should be very careful in 
interpreting the results, as only vety large impacts are likely to be judged 
as significantly ditTerent from zero. As a result, such states might 
consider pooling smnples over time to iticrease sample size and thereby 
enhaiKe tte reliability of tte net imj^t fmdings« 

Finally, although a total analysis sample size of 12,000 should be 
^lequate to meet most states' needs, tte aj^rc^ate sampte size expends 
on several factors and there Ls no size thm is correct tuKfer all circtmi- 
stance. States that are unsure as to the appropriate sample sizes to use 
in a n^t imp^t analysis ^ould discuss tl^ir concerns with a sampling 
expert. 
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I>aia C^Mectian and JRrocessing Plan 

A variety of data collecti(m and prwressing tasks must be conducted in 
suppon of the overall rcse^uch design. Quarterly samples of particiiKints 
and comparison group members mast be drawn. Preprogram, program, 
and postprogram data must be obtained from JTPA, ES, UI, ami PA 
(MIS) records. These must be merged with individual participant and 
comparison group records. All data must be cleaned^ and certain cases 
may need to be excluded. Analytic variables must be specified, and 
procedures must be implemented to deal with missing information. 
Finally, analysis files must be developed. This section reviews the 
various data collection and processing tasks that must be conducted. 
Readers interested in additional detail arc referred to the implementation 
guicfc on net impact evaluation in the series of evaluation guides listed in 
the appendix. 

Although nmc of the data collection and processing tasks outlined in 
this section is particularly difficult, the overall magnitude of the under- 
taking is consi(terable. Moreover, there is a major ccxmiination issue, 
since many of the tasks must be performed by staff of several different 
agencies or subagencies. Tte size and breadth of the data-rclaied tasks 
have two important implications. 

First, there must be active and continuing cooperation and support on 
the part of several state agencies and subagencies. Because these agen- 
cies have different policies and priorities concerning issues such as data 
confidentiality, any issues of ccmccm must be resolved at the outset. A 
regular data collection and processing schedule must also be established. 
Lack of support on the part of any of the agencies involved will consid- 
erably reduce the value of tte net impact results and could rencter them 
useless. 

Second, one person should be given the responsibility of managing the 
full range of tasks and the authority to obtain the neass^ staff and 
computer assistance, TTie magnitude of the data processing tasks and the 
involvement of multiple agencies n^ke li:^se conditions particulariy 
important for managing the data processing system* Although managing 
this effort is likely to be a timc'^^nsuming activity in the initial stages, 
once tl^ syntem is in place and the individual tasks become routine, the 
management time required will decline considerably. 
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Table 2 A shows the various data collection and pnxcssing tasks 
involved, from sample selection to preparing tm analysis file for estimat- 
ing Ite net impacts of JTPA. To increase clarity, we have chosen to 
present as separate steps some activities that a)uld easily be combined 
into one step. In addition, note that there are likely to be imanticij^ted 
data-related issues and problems. To minimize such problems, it is 
j^icularly imjxjrtant for researchers and key data prcxressing staff 
members from each of the involved agencies to m^t frequently. Fre- 
quent communication helps identify idiosyncracics in the systems, 
which could produce nwicomparable data for certain types of individu- 
als. This communication may produce potential solutions for such 
problems as well. 

T8llt8 2.4 

(tosnri» of Data Cirih^teii Procttting Ti^ 



» Setect qi^rteriy sampler of JTPA participmits and obtm, rnerge, and prtx^ preprogram 
from various sources for th^ paiticipants. 

• Sel^ expanded qi^rteriy samples of new ES regi^rants are ^UMiomioiily disadvantii^ to 
serve s& comparison group niters. 

- Obtain, mer^. and procef^ prepn^ram data from various soun^ for the exjmndad samptes. 

- Setect quaiterty compart gnoups of aduK men and women ES r^imrants from the expamted 
samples to mateh tt^ distrit^ution of partrcif^nts on four cel& compri^ng combinations of Ui 
recipri^ ^us and AFDC r^pient ^atus. 

• Merge tt^ quarterly prqirogrw dala f ties ttet irK^iude ^ d the data ot^taioed In the above steps for 
the sample of partteipams and compart group members. 

• Cr^ ^parate annual preprogram analysis fifes for suJutt men and women from tf^ quarterly 
pfeprpgram data files. This invoh^ mer^ng the quarter^ fiies. editing the data, creating ar^^ 
var^tes. and imp^menting proc^ures to f^dle c^es wWi mts^ng data. 

• Obtain prc^ram and pc^rogram outcome data (i.e.. AFDC Grante Recmds and UI V/s^ Records) 
f(^ all qi^rteriy samp^ of particq^^ and comparis^m groiq? numbers and cr^ ^^)fopftate 
outcome variables. 

» Obtain data on JTPA sef^m:es for par^ii^nts in ^h of ti% quarterly samples and cr^ ^rojmte 
at^hrity or service variab^. 

« Cf^ se|^teanmiair^im{»d analysis fi^ for ^uit men and wo^ 
pn^rogram analysis flies with the Du^n% and treatment var^btes. 
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Selecting Partidimnr Groups 

TTk? first data collection and processing task involves participant 
group selection. As noted above, samples of adult men ami women JTPA 
Title II-A enrollees must be selected on a quarterly basis. In table 2.5, we 
list the steps used to select appropriate f^icipant grtHif:^ for a particular 
quarter. Minor modifications may be made to accommodate states that 
desire larger samples. 

Although the procedutes listed in table 2.5 could be used to select 
participant groups in any stale, some of the steps may not be necessary 
in certain areas. Some states may need to alter the procedures slightly to 
n^t^ir needs. For example, because tt^ statistical precision of the net 
impact analysts is not very sensitive to moderate sample size differences, 
given the large sample sizes involved, a state that serves adult men and 
women in JTPA in approximately equal proportiwis could omit the step 
involving sample stratification by gender. However, because minimal 
effort is required to stratify the samples, and the models are to be 
estimated sejmrately by gender anyway, prestratification by sex is 
recommended. 

The fourth step in this list concerns the procedures involved in 
selecting the participant samples. Specifically, we suggest that a range 
of twcMligit numbers {(X)-99) be specified (with the size of tte range 
dependent on the sample size) and that the last two digits of the person's 
SSA number be used to select tte sample, since it is a random number. 

T8ftliZ.5 



1 . CfiMdB a file of al^ ^^^m wtm ertrc^W m JTPA Title If-A programs in any SDA during ttie quarter 
that inclu{te SI . jtnber, ^b. ma sex, 

2. E«dud6lln>sei^areuj^rc;ge22ofOs^^64{orper^ 

3. Create separate subfiles for aduft men and women. 

4. Setect a ramkmaampie of aMnm and won^nfnmi the two strata subtile 

digits ot fl?f SSA nimrlwr. «rfrtch are ramton numbers, Tfw size of tite Quarter^ sample should 
seasonal drner^^kiemttfrnemamibeaachtiiaitfteMan^^ 
^er exduding cases tor n^irg data. ir^iOs at ^ 3.(XX) adtdt rrm and women each. 

5. Obtain pre^m^ram PA Grants Ft^nte ami Ut tomtit History <tito and cr^ measures of fif DC 
recipient status and Ut r«:ripi«i! st^ at ^(^Iment . 
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ftily individuals whose SSA fina Wigit numbers fall into range would 
be included in the sample. In addition* to account for potential '^easonal 
differences in JTPA enrollments, one should select a fixed proportion of 
enrollees in each of the four quarleni, using data on enrollments in the 
prior year to determir^ appropriate proportiOTs. This is suj^rior to 
selecting an equal number in each quarlen-- 

Given an estimated 5 percent j^mplc loss due to missing data, the rec- 
ommendation in step 4 translates into the iniiial selection of approxi- 
mately 3*200 adult men and women each to yield an analysis sample of 
3,000 each. States with more severe missing-data problems would have 
to select larger initial samples. Finally, in step 5, key preprogram data 
are obtaimi to use in developing the matched compariscMi groups. Tlie 
data sources to be used and the specific measures to be developed are 
discussed below. 

Selecting Comparison Groups 

As described earlier, the comparison groups are developed from new 
ES registrants in areas served by the SDAs. Table 2.6 provides an 
overview of the steps that ES data processing staff could use in selecting 
appropriate comparison groups of adult men and women for a particular 
quarter of JTPA enrollees. Refuting these steps for the subs«}uent three 
quarters yields matched comparison group members for the entire 
program year. 

Tte first four steps listed in table 2.6 are designed to yield a sample 
frame that is son^what more comjmrable to JTPA i^ici{^ts than a 
sample of all ES registrants. Steps similar to tl^se would also be used to 
deveico matcted comparison groups. TTiese initial steps mirror the first 
three steps for selecting tl^ particii^t samples. 

ITie fifth step addresses tfie need to expand the initial sample of ES 
registrants to overcome the s^ple Ic^s expected at steps 7 and 8. 
Specifically, the size of the initial samples must be large enough 
ultimately to yield suflficient numbers of ES registrants w1k> have not 
participated in JTPA in each of fcmr c^Us defined by combinaticms of UI 
rreipient status and AFDC recipient status. This assures that a ccwnpari- 
son group with a similar distribution of these characteristics can be 
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Takli2.6 
to tetettof tk$ COL..»iiim &mrp 

1. C^i^t^ofall ESoHices m areas s^ved by SIMs m the st^. 

2. Create a fUa of ail pmot^ who u^e tmw r^ismmte in ^ise ES ofhi^ during the quarter that 
indt^ SSA mrmber, sex, and ^OfeXhm tl« pers(^ ecomm^^ dsadvanti^. 

3. Exchotettosewtoare: (d}n{^«^cHn^di$^nt^.(b)a^21 ort^,or(c)wfK)are(rider 
than the oMti^ Imtlvkliml tn the quarter^ JTPA ^roll^ sampte. 

4. Create ^Kirate^iMUestrf^idtn^ and mn^. 

5. StmH an expamSed ramtom sanr^ of aduit men astd worm new ES regtstiants from the two 
s^iarate sMm, As a ^neral guHtetit^, aj^roximate^ 2JSSXi of aduft men and woman 
quarter sh«j]d be su^:mnL 

6. Obtain avaiiabfeM^GfBfHslteconte end UJB^iemH^^ 

recipient stahjs and Ulrec^^stsrtus at regi^nts ktentified in step 5. 

7. E)ChJdepers(m8WfK>8f«curTentlypartW{atif^jnJ^ 
year ba^ on a n^dch of SSte. 

8. SetecteeparateramtomsamplesofadiMmenandvi^nmth^n^tch^ed^bu^ 
on tf^ f<MJ r OHnpri^ the ctmibina^ of ^ comp^ 
itfDCfBCii^ntsteti^ 



drawn- Because the exf^nded initial sample size is likely to vary 
ccmsiderably from state to state copending an individual characteristics, 
local eccmomic ccmditions, ami the state policies concerning tte relation- 
ships among the local ES, AFDC, and UI offices, it is very difficult to 
provide jOTcisc guidelines. As a stalling point, we rmnnfliend initial 
samples of 2,500 adult men and 2400 ^ult women ES registrants be 
selected in each quarter. 

Hie mxl step involves obtaining certain preprogram agency data aixl 
(teveloping measures of AFIX! recipient status ami UI claimant status at 
enrollment. AfterexcludingcunentorrecenlJIPAi^rUciimtsfiromtte 
comparison group sample frame, based on matching SSA numbers (step 
7), the final step involves allocating the remaining SSAs to the four 
recommended stratincation c^lls comjmsing combinaticms of UI r^pi- 
ent status and AFIX: recipient status. Sampling rates for each cell would 
be used to match the quarterly dis^bution of i^cipants m tl^e key 
char^^ristics. 

TTiesc procedures can be used by most states, Init scmie of tlw sp^ific 
steps may i^d to be modified to meet various states* analytical t^eds^ 
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data limitations, or specific circumstances. For example* significant 
modifications to the procedures may be necessary in states that do not 
have all of tlK required data elements in their ES MIS system, in 
particular, data on wlK^ther an ES registrant is economically di^dvan- 
taged. Because being economically disadvantaged is the primary eligi- 
bility criterion for JHTPA, it is important to develop procedures to t^lcct 
comparison group membets who arc also economically disadvantaged.^^ 

In the absence of specific infant 'ion on economically disadvantaged 
status, an alternative appTX)ach. used extensively in the literature, in- 
volves excluding individuals with very high preprogram earnings and 
then explicitly matching tl^ remaining comparison group membet?* to 
participants on the Imis of preprogram earnings. That is, instead of 
excluding all persons whi> arc not economically disadvantaged at step 2, 
one would first obtain UI Wage Records for a much expanded sample at 
step 5 — perh^s up to five limes as large if only 20 percent of ES 
registrants are economically disadvantaged. Tlien, exclude all persoas 
with high earnings in the immediate preprogram period, who would 
certainly not be eligible for JTPA- The precise cutoff level is a matter of 
judgment and depends on the distribution of preprogram earnings in both 
samples. As a general guide, however, a cutoff level set at the maximum 
earnings of participants (separately for adult men and women) in the six 
months before enrollment or somewhat higher (for example, one stan- 
dard deviation), seems reasonable. 

A final issue in selecting tte matched comparison groups involves the 
development of ccmsistent measures of AFDC lecipient status a'Hl UI 
recipient status at enrollment. B^ause ES and JTPA data bases will not 
generally collect comparable data on these two factors or maintain tte 
enrollment values in the MIS, one must develop consistent me^ures of 
these items from the f^me data set in order to deveiq} appropriately 
matched groups. The recommencted appro^h for developing tl^se 
measures is discussoi below. 

Obtaining Preprogram Data for Participants and 
Comparison Group Members 

An integral step m selecting matched comparison groups for a specific 
program quarter of JTPA Title II-A adult men and women participants 
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involves the processing of several different data elements from agency 
records. In addition to JTPA and ES enrollment/registration data, AFDC 
Grants Payment Records, and Ul Benefit History Records, it is imjwrtanl 
to ohmn Ul Wage Records for the preprogram period soon after the 
participant group is selected. Timeiy acquisition of these data is neces- 
sary because ( 1 ) some states do not retain much historical MIS data, thus, 
the sowier the data are obtained, the longer the preprogram period 
covered: and (2) some of these preprogram data items are used to develop 
a proHle of individual characteristics that in turn is U5^ to select 
comparison group members similar ir these characteristics. 

As a result, though the steps for obtaining data for the preprogram 
period for comparison giuup members are identical to the steps for JTPA 
enroHees, the liming and magnitude of the task differ considerably. TTic 
various d ita elements to be obtaineu and meiiged with the quarteriy 
samples of j^rticipanls and compar son group members are described 
below. Some problems that may be encoimtered in this process are also 
identified. 

As indicated in table 2.7, the first two j^ts of data elements for 
individuals in the analysis come from JTPA applicaticm information and 
il^ ES application form. Although only a few items from these forms are 
used in support of the steps listed in this table (e.g,» age, sex, disadvan- 
taged status), it is useful initially to extract all application data from the 
JTPA and ES MIS systems for thc»;e imiividualL select into tte quarterly 
samples for some analysis purposes. Although only the data items that cdn 
be regarded as ccmiparably measure for participants and comj^son 
gnxip members will be used for net impact analysis, all JTPA application 
data should be obtain^ in ca^ states are intere^sted in using the nwt impact 
design to estimate grc^s program impacts, or to examine whether assign- 
n«nt of program ^tivities (services) to individuals depemls on other 
measured characteristics avail^le for participants. Morw)ver, all ES 
application data should be obtaine to get a better sense of tl^ character- 
istics of this group and tK>w tl^y are likely to differ frcmi the characteristics 
of the individuals in ih^ participant group. 

Detailed preprogram Ul Wage Records ar.d AFDC grants histories are 
particulariy important io a net impact analysis. Ideally, three years of 
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Table 2.7 

(Atalolm ^dteaftn and Kter ftsi^sgfan Data 



1. Obiajn JTPAenrolftTwnt jrtormafion for eadt taiticipam: igefstst in sheeting sampte (see 

1-4 of ^ 2.5); (rt»w itens iS8d to ite^^ varte!^ tor net imjs^ ana^ 

2. OtMn ES r^^tiation M for euh potential compart group n^t)ef. Ag&. ami 
Bcwofrt^ (tead»«rt^ state used hi devriojHng franw for se 

{sw 2-4 (rf tal^ 2.6); otfwr ttenw to (tevelop wrlalrfes for rot fnnsrt ai^^ 

3. Obtain jHeproflraninwntWyAFDC Grants Rewrite for up to ffiree years tor partte^^ 
afte'8t^4oftaWe2.5andfor»mparteon group fneinb«sictenMfi«lln^5oftal*2.8,and 
ciaati (a) vai^tes me^ring (HarterJy totate and (Ij) an AFDC rwapiwit-st^ IfHlteator, (tefirwJ 
« 1 the persOT recehred WTK; graiits h) tt» riKwft before, duri!^, or after the mt^ 
enroflment w q^kst^, ami 0 (^mnse. 

4. Obt^UltoiemHfi*My data f«H»(^jMfar quarter before araJ after enroHnwrtftff 8^ 

to step 3 abow, ai<d cjwte (a) ti«il Ul toneBs recwred durtrio Mie quarter prfcff to KiroBnwrt 

awjUcatton ami (b) a Uf reclpienl-^ttis imitator, (tefined 

b^wie.durit^ or after tf»nK>ntti(rfenntfnrK)twaw)lic^ion, and Oothenv^. 

5. WtoJ up to 12 gifflrtOT of prapn^ramUl Wage Records, mdcra^ftaals^ 



preprogram data are n^ded. TTierc are gei^rally six to 1 2 qumers of UI 
Wage Records available at any one time, with approximately a three- to 
six-month lag before these data are complete. As a result, to ensure that 
wages for the immediate preprogram quarter are complete for the 
analysis, it is necessary to update the data for this quarter when the 
posQ)it)gram eairongs data are ol^aii^. 

(tee must also obtain AFDC grants rwieived by participants and 
«)mparis<»igroupmembers. InacUid(MitooNaining(»^^)rogramm(»ithly 
grants records for up to three years to serve as important cwitrol variables 
in the net impact modelf., in order to <tefine welfare-recipieni status 
similarly for JTPA ptiicipaius and ES re^strants, data on AFDC grants 
received for iIk month after enrolbnem month must be obtained for all 
individuals. To minimize problems caused by differences in the length 
of time from JTPA or ES application to AFDC enrollment, w caused by 
differences m recording practices among different agencies, a JTPA 
enroUee (or ES registrant) should be defined as an AFDC lecipicnl if the 
person received ARDC grants during the calendar month prior to enroll- 
ment or registration, the month of enroUment or registration, or during the 
month after enrollment or rcgiaration. Similariy, a UI- recipietwry-status 
indicator can be develqjed using the same aj^roach with data from the 
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UI Benefit History file. Although these definitions may ditVcr from 
JTPA or ES definitions of welfare-recipient status or UI claimant status, 
they will enable one to generate a comparison group that is statistically 
similar to the partici}Mit group on this important characteristic. 

The PA MIS systems in some states may present obstacles to obtaining 
suxurate preprogram AFDC Grants Records for certain types of indi- 
viduals. For example, in anempting to develop prcpn>gram measures of 
grants ^d, because of changes in household status and other factors, the 
data base may not allow one to verify that a specific person wie^ in a 
particular assistance unit throughout the three-year pericKi. This may be 
true even though it may be jxjssible to identify up to three prior years the 
prepnsgram monthly grants paid to a particular assistance unit for a given 
individual currently in that unit. As a result, the preprogram AFDC 
history for that unit may not accurately reflect a person's welfare- 
recipiency status during that period. This is particularly a problem for 
individuals who experience a marriage or divorce, or who change living 
arrangements. 

To obtain accurate information on the preprogram AFDC status of 
I»rlici|»nts and compari.son group members, such problems must be 
overcome to the extent possible. A potential solution available in 
Washington State would involve using the "Warrant Roll Extract Files," 
which contain a record of all AFIX: payments m^e each month, and a 
list of all SSA numbers in \he household that month. These files could 
be linked over time to determine which assistance unit the person of 
interest was in, and to obtain the correct preprogram ^ata. 

Preparing a Preprogram Analysis File 

TTie next task is to develop preprogram analysis files— -created from 
the annual preprogram data file — for adult men and women. Once the 
preprogram analysis files are ckveloped, the analytic procedures de- 
scribe in later sections can be implemented to investigate the compara- 
bility of the JTPA enrollee and comfmrison groups in the preprogram 
period. 

Before describing issues involved in conducting these tasks, one 
important feature of the preprogram analysis file shcHiid be noted. 
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Specifically, because preprogram daia elemcnis are defined in terms of 
their relationship to the quarter of enrollment, elements in the same fields 
on the analysis file will correspond to difTerent vak9Hlar perknls for 
individuals who enrolled in dififerent qumteily files. For example, data 
elements for the immediate preprogram quarter, on the file for the 
enrollees in the fir^t quarter of a program ye^, will correspond to the 
period of the second preprogram quarter for individuals who enroll one 
quarter later. Befoitr comparing dollar amounts in certain preprogram 
quarters across files, therefore, one must adjust for overall price changes 
by translating all nominal dollar amounts into real terms.*^'^ 

Editing data fdes. Although cT>nsiderable cleaning and (^ting will 
have been performed by the respective ES and JTPA data processing 
staffs as part of their normal procedures, one must ctmduct edit checks 
to become familiar with ditlerent files aiKi tocheck data quality. The first 
type of edit check compares iIk results of a simple fr^uency distribution 
on all variables in each of the annual files with a range of acceptable 
values.^ For other data elements, such as UI wages, Ul benefits, and 
AFDC grants, a range of acx^ptahle values should be created that 
incorporates rough estimates of the maximimi amounts that can be 
received from certain programs in the slate. 

Although some errors are obvious by inspecticmi of a single data item, 
other erroni may nol be apparent except when viewed in combination 
with another data stem. Tlius, as a second edit check, limited cro^% 
tabulations must be carried out regarding certain items, to identify 
additional potential data quality problems. For example, cross talnila- 
tions of age by education could identify 22-year-oId individuals with 19 
years of education, an unlikely occurrence. \i is also useful to cross 
tabulate earnings and AFDC grants received an the same preprogram 
quarters. The preseiKe of individuals with large values for earnings and 
AFDC payments in a given quarter may be indicative of data errors or 
other problems. 

Specifying analytic variables. The analytic variables specified should 
be comparably measured for enrollees hikI comparison grmip members. 
For variables derived from a common source, .such as AFDC Grants 
Records, UI Wage Records, and UI Benefit History data, comparability 
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should mn be a pmblcm. However, measures of persona! characlerisiics 
wJH be obtained from berth the ES and JTPA MIS. and differences in the 
ways in which qucHlions are asked or answers arc recorded can present 
major obstacles to defining comparable variable deflnitions. Morcoven 
even when questions and respon?^ codes appear to be the same, the 
information collected may correspond to slightly diflerent concepts due 
to diffcrciKCs in staff instructions and ti^ining. Because of these 
potential problems, one must review the application forms to both the ES 
and tte JTPA and the corrcsiwnding haiKlbooks that provide instructions 
for recordiiig answers to each question, and resolve remaining issues 
through discussions with appropriate agency staff. 

At a minimum, one should develop comparable measures of age in 
terms of years, a set of dummy variables for racc/elhnicity and sex, a 
dtmimy variable for veteran status* and a set of dummy variables for 
occupation, based on the first digit of the DOT code. In addition, qt^ 
should develop limited indicators for educational t^ckgrpund that are 
comparably measured. For example, the ES application fomi in most 
slates generally collects education infomialion in tte form of the highest 
grade of schcxiling completed (from 0 to 19 years); whereas, the JTPA 
application form often recx)rds an individuaPs educ^ation status in temis 
of one of the following four codes: ( 1 ) school dropout, (2) in schcHil (high 
school or less), (3) completed high schtx)! or received GED, and (4) 
currently attending or has attended schooling programs beyond high 
school. With such information, however, it should be possible to recode 
values from the ES application form to specify separate dummy variables 
for whether the person is (a) not a high school gnuiuate (i.e., ()-l 1), (b) 
a high school graduate (i.e., 12), and (c) has completed additional 
schooling beyond high school (le.. 13 or more). Every effort should be 
made to implement procedures such as these wbewver jK>ssible to define 
comparable measures of variables for both groups. 

Procedures for fuiNcJling cases with missing data. In general, tte 
variables used in the analysis should not suffer fmm major missing data 
problems. However, in instances where independent variables (e.g., age, 
education) have missing values, it is prefemble to adjust for the missing 
variable in question rather than exclude all cases that have missing data 
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on any relevant variable. Although there are seveml alternative proce- 
dures that one can use to create substitute values for missing data, the 
gains from using an elaboraie system to fix relatively minor problems is 
not likely to be worthwhile.'^ 

As a result, the mean values of the independent variables — calculated 
.sefKuately for fwticipants and comparison group members, and, of 
course, separately for adult men and women — should be used for cases 
with missing data. Thus, as part of the initial analysis task, one should 
calculate the means of all independent variables separately for partici- 
pants and comparison group members on the analysis files, and ptejmre 
recode statements that set the value of a variable equal to the appropriate 
mean whenever it is missing. In addition, if differences in the independ- 
ent variables among the quarteriy samples are likely to occur, one should 
consider using means calculated separately by quarter of enrollment or 
registration to capture trends in these variables over time. The treatment 
of missing data in the program participation or service variables is 
discuss^ below. 

OlHaininfi and prm essing ditrin^-profinmi and fH}stprogram out- 
come data. Postenrollmeni UI Wage Records and AFDC Grants Records 
must also be obtained for both the participant and comparison groups. 
Hien, appropriate variables must be specifiwi and merged onto the 
preprogram analysis file. In addition, UI Wage Records must be 
obtained for the immediate preprogram quarter for all participants and 
comparisi, .j group members to correct for potential measurement error 
problems due to obtaining data "ttx) early" for that period. 

No problems are anticipated in collecting quanerly UI Wage Records 
in the jxj&tenrollment period. The necessary information can generally 
be obtained from a single request made at the very end of the data 
collection process. In states that retain only six quarters of UI Wage 
Records, m intermediate request must be made no later than 18 months 
after the month in which the firel quarter of individuals were enroll*^, to 
ensure that the entire history can be obtained for early enrollees. 

With regard to collecting monthly AFDC Grants Records during the 
program and postprogram periods, it may be preferable to obtain such 
information on an ongoing basis, rather than only once at the end of the 



ERIC 



1} \) 



Evaluating Net Program Impact 87 

data collection proccKs. In working with PA data systons in which ite 
Recipient History File docs not eimble chic to icfentify whether a {^rson 
is in the particular assistance unit thriHighout tl^ period of interest, it is 
preferable to obtain the information on an cmgoing basis each mmth to 
iriiiiinii2£ measurement error. Obtaining informaticm on this basis 
rcquiies timX fen- e^h of the subsequent months tte SSA numbers of 
participants and comparison giT>up members must be compared with the 
list of SSA numbers in assistance units th'tt rrceived AFDC payments 
during the month. Tte actual values of monthly grants would be included 
for tl^ SSA numbers that matched, «uid zeros would be included for those 
SSA numbers that did not match, (^arteriy am] annual values would 
then be calculated as tte sum of monthly values. 

Note that by stewing the collection of tl^se agency data at a single 
pomt, one will obtain eight quarters of UI Wage Records and AFDC 
Grants Recoitis for participants and comparison group members who 
enrolled or registered during the first quarter of a progiam year, seven 
quarters of data for thoj^ who enrolled or registeied during the second 
quarter, six quarters of data for those who enrolled or registered in the 
third quarter, and five quarters of data for those who enrolled or 
registeied during t)^ last quarter of the program year. Each set of 
quarterly earnings and AFDC grants data would include one quarter for 
the actual quarter of enrollment or registration, two sut^equent quarters 
of data that are likely to include program earnings for some participants, 
and varying postprogram quarters of earnings r^OiXis. Because of the 
usefuhress of having rectangular analysis files, a common number of 
postenroUment quarterly values should be created for all individuals 
onhe file, and milling data codes (e.g., -9s) pl^ed in postprogrum 
quarters for later enroliees for wlK>m data are not yet available. 
could, of course, subsequently obtain actual values for tl^se quarters and 
replace the missing data codes. 

Ohtaining and Processing JTPA MIS Data 

Because individuals can participate in multiple activities, most state 
JTTA data systems will have a iwogram ^tiviiy file in which a given 
iiviividual may have multiple records. To sprcify ccMisistent analytic 
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variables, program €K:tivily records must be extracted for all SSAs in the 
four quanerly samples of JTPA *^ millees, Dei^nding m the archiving 
procedures followed in a given state, it m»y be possible for ttese data to 
be obtained from a single request made at the end of the data collection 
process, and hopefully after ail or almost all individuals have terminated 
fn>m JTPA. 

After the program activity records arc obtaim!, the next step involves 
specificaiicMi of variables to represent .wrvic^s received by JTPA partici- 
pants. In general, it should be possible to develop relatively detailed 
indicators of the services received from JTPA, related to the type of 
occupation and length of tmining, and wtetl^r the person completed 
training. These variables an: described in iIk conceptual framcworic 
secticm. Once ttese variables are specified, it^y should be merged with 
the preprogram analysis files for participants, and zeros must be entered 
for comparison group members for all of these variables. 

If participants have missing data (m key JTPA treatment variables, 
they should be excluded from the analysis samples, provided the reason 
the items are missing appears to be random (i.e., not systematically 
related to the likely net impacX of the program). To make this dctermi- 
naticm, one must compare the characteristics of partici{^ts who have 
missing data on the variables (c.g,, length of participation) with the 
characteristics of participants who have data on tl^ variables. For 
example, one should compare the age, race, and education of individuals 
in the two groups of participants to determine if there are major differ- 
ences, 

Morci^ver, if in the process of collecting data on program experiences 
one also obtained information on placement status at termination, it is 
useful to compare JTPA enrollees on their placement status at termina- 
tion to judge whctter having missing data is systematically related to the 
impact of the program. If enrollees with mis.sing data on JTPA experi- 
ences are equally likely to be placed in jobs following the program as 
enrollees with complete data, this would provide a<kiitional confidence 
that the validity of the analysis will not be compromised by excluding 
such individuals. 

Once the JTPA analytic variables have been developed and decisions 
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msKlc on tte trcatment of misning dala, iho final lask involves merging 
these variables and the during-pmgram and posiprogi^ outcome vaii- 
ables to the pn^program analysis files for coch individual by SSA 
number. This results in the creation of net impact analysis files for adult 
men and women. TlKrse files will support all of the analysis tasks 
described in the next section. 

Data Analysis Plan 

A data analysis plan must be developed to examine the adequacy of the 
comparison groups selected, and to use the comparison groups to 
estimate the net impacts of the prognun on the outcome measures 
specified in the conceptual framework. This section pirscnis an overall 
strategy for obtaining valid estimates of the net impacts of JTPA 
prognuns on the posiprogram earnings and welfare dependency of adult 
men and women enroUecs. 

Before describing the cfetails of the plan, we want lo emphasisa: that the 
iwommended approaches should be quite accessible to all staivs inter- 
ested in conducting i^l impact analysis of JTPA or of other social 
programs. For example, all of the analysis techniques to be used are 
c^tained in standard statistical software packages — such as SAS and 
SPSS — that should be ivadily available and familiar to state-level 
analysts. In addition, aHer some initial data processing on a mainframe 
computer, it may be possible to download the analysis files to hard disks 
that can Inr accessed by minicomputers. This will minimize the computer 
Itrsource^ required to conduct the analysis. 

An overview of key elements of the data analysis plan discussed below 
is shown in table 2.8. We first describe an analysis strategy for 
examining the adequacy of the comparison groups selectwl in order to gel 
a better understanding of the direction and magnitude of potential 
selection bias. We then describe a statistical model that can be used to 
estimate the average net impacts of JTPA and the impacts for imponant 
subgroups. The section concludes w ith a discussion of potential adjust- 
ments for ceilain data and design deficiencies. 
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Examining the Adequacy of the dmpahson Groups: 
Obtaining Evident e on Selection Bias 

If the samples of JTPA participants and ES registrants developed 
through the (^tef» described in the previous sections aiie similar on both 
measured (e.g., age, race, education) and unmeasured (e.g., aUitude 
toward work, motivation) chiuacteristics, then valid inferences about the 
impacts of JTPA programs can be drawn from such comparisons. 
However, whether an individual participates in a social piDgram is likely 
to depend on both individual and agency decisions. 

TaMi2.« 
Dvenrttir of Dila toatysb Ptas 



• Examine the adequacy of comparison groups using armlys^ techniques such as difference in 
means, difference in distrtbufions, and muffipie n^ression ana^, The adequacy of the 
company groups wiH be jud^ m terms of three criteria: 

1 . Similarity of participant and compansiKi grwps on nreasuied character^ics (e.g.. a^, race, 
^ucation). 

2. Similarity of i^rtcipant and comisrfson groups on preprogram ^mings and AFDC grants. 

3. SimiJarity of the retation^ips betwwn preprogram earnings {mi AFOC gtanfe) and measured 
Ghar^erist!CS tor partici^nis and con^r tsm group members. 

• EstimateavrersqjenetinipdCtsofJTPAforaduKn^nandwDmenus^ Net 
impacts will be e^imatwl for four postpn^ram outconw n^ures-HKimings. whether employed. 
AFDC grants, and ^t^hm an AFDC recij^nt— in e«!h of three different pc^program ^toJs: th w 
months, six monfi^. and 12 months. 

• Estimate net impacts of JTPA m tte vaifeus outcome measures for adult mn and wwnen and Jwy 
subgroups i^ing autoregr^sh^ models. In addition to sex, the ^bgroups of interest iiKiude: 

1 . f^rticipdnt characteristics such as r^ or ethnicity, ^e, «lucat<on. and welfare t ecipfent status for 
women. 

2. Program acthrities such as CT, OJT, JSA. and all other activities. 

3. Program length of stay. 

• Adji^ net imp^t ^imates to ^ extent pebble for data and design deficiencies: 

1 . Contamination of the con^rismi groups. 

2. IMcoven^ eamii^. 

3. Selection bias. 
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For example, JTPA participants must deci(k to apply to the [m>gram«, 
meet certain legislated eligibility criteria* be selected by tte agency for 
program participation and assignoi a program t^tivity, and decide to 
accept that assijpiment ami enroll in tt^ program. Although ES regis- 
trants do not have to meet any formal eligibility criteria, certain individu- 
ids, such as those receiving benefits from government programs such as 
Vh are required to negister with tbe £S^ ami some offices follow selective 
registration policies. Furthermore, whether an £S registrmit recei ve!i ES 
services (kpeiKls on several factoiii. including the availability of suitable 
job openings, and the person's qualiftcatioas and persistence. Because 
of Uwsc various selection processes, it is unlikely that the resulting 
samples of JTPA enrollo^s and ES registrants who do not receive 
services are truly equivalent on both measured and unmeasured charac- 
teristics. This is tte issue of selet thm bias, which is highly likely to be 
present in ev^uations of other social programs as well. 

All mmexperimental appit^hes to evaluating the net im}^t of a 
social program will probably contain a certain amount of bias. Hiat is, 
the formal conditions requhied to ensure unbiased estimates of program 
im}»cts are not likely to be met, even if one had extensive data on the 
characteristics of progt^ enroUees and compariscm group members. 
This is true for the proposed research design. As a practical matter, 
therefore, cme should not focus on the fact that the two groups are not 
identical, but identify tte major din^^nstons on which the groups differ 
and determine the extent to which the net impact estimates are likely to 
be sensitive to those differences. 

As indicated in table 2.8, three diHerent criteria can be used to judge 
the adequacy of the comparison groups selected: 
I • Similarity of the JTPA enrollee and comjwison groups on measured 
imlividual characteristics. 

2. Similarity of tte JTPA enrollee and comparison groups on prepro- 
gram earnings and AFDC grants. 

3. Similarity of the relationships between preprogram earnings (and 
preprogram AFDC grants) md the measure imlividual characteris- 
tics of JTPA enroUees and comj^son group members. 

Although these are the traditional criteiia forjudging the adequacy of 
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nonexperimentally (krived comfwisim groupn^ th^y are necessary* but 
insufficient, conditions for overcoming .selection bias. Even if the 
compariHon giDups selected generally meet these criteria, this shiMild not 
be interpieted as dcfinilivc evidence of an absence of selection bias. 
With this caution in mind, some analyses are mitlined below that can be 
performed for each of the quarterly samples, and for the annual sample 
as a whole, to see whether these criteria an! met. If they are not met, the 
analysis identifies the types and extent of ditTerenccs between the 
groups. H^se factors must then be kept in mind when interpreting i^t 
impact results. 

The first criterion (Criterion I ) is the similarity of ll^ two groups on 
measured characteristics at enrollment or nrgistmtion. It is imrticularly 
important to compare the particifmt and comparison grou}^ on available 
measured characteristics known to aftcct clings and AFDC grants. For 
example, it is particularly useful to determine to what extent th^ two 
groups differ on age, race, education, occuf^ticm, and handicapped 
status, and other relevant personal characteristics that are comparably 
measured for both groups. 

Using standard softwaie packages, one would compare the means mi 
the distributions of these measured characteristics for participants and 
comparison group members (sepanalely for adult men and women) in 
each of the fou*- quarieriy samples and in the overall program year 
sample,^ Because the output from standard software analysis pi^kages 
normally includes the results of Me.sts Chi-square tests for formally 
testing the equivalence of the means and distributions of variables in two 
samples, it is straightforward to compare the similarity of the participant 
and comparison groups on all measured characteristics. 

Similar analyses should be coiKiucted across JTPA program activities. 
That is, one should not only compare the chanicteristics of participants 
to the characteristics of comparison group members, but also comj^rc 
tl« characteristics of j^icipants with respect to program activities 
received, such as CT, OJT, and JSA. This will indicate any additional 
selection bias arising in estimating i^t impacts by separate program 
activity. For example, if one determiml that more motiv^ed or 
energetic individuals weit? being sent to OJT» the net impacts of this 



Evaluating Net Pinc^ratti Impact 93 



program ^ivity would be Kon^wlm inflated becau$^ of this asstgniiKnt 
procef^K. On t)^ ott^r hand, if there weie relatively few differences in 
measured characteristics by progrmn activity, this evicknce would 
provicte some confidence that no additional selecticm biases would be 
introdiMred in deriving estimates of tte net imfmcts by pn^rmn miiviiy. 

The second criterion (Criterion 2) to judge the adequacy of ite 
compi^son grou^Hi is the similarity of the key outcome measures of 
participants and comparison group members in the pnqprogram period. 
TTiis involves a test of wheiter a significant difTer^tce exists in the 
preprogram earnings and AFDC grants of the two groups, ctmlrolling for 
measured characteristics. Such a test provides valuable evicfenc^ on 
whetJKr tl^ two are ccmiparable on tlK^ basis of tt^ lagged ikpendent 
variables or, in other words, whether there arc differences in the outcome 
variables between tlu; grcHips in the preprogram fH^ricxi that are due to 
tmmeasured characteristics. 

If there are any differences in adjusted preprogram earnings or AFDC 
grants between tl^ two groups, then this analysis will also provide 
evidence as to tl^ direction and magnitude of the selection bias. For 
example, the extent to which JTPA participants have larger (smaller) 
adjusted preprogram earnings than £S registri^its provides some imlica- 
tion as to whetl^r they are more (less) advantaged on the basis of 
unnieasured characteristics. Moreover. iJw size of tlw estimate differ- 
ence is a reasonable estimate of the amount by which the i^t program 
imp^ts could be oven;tated (understated) if the difference persisted in 
the postprogram period. 

To formally test for differences in the preprogram earnings and AFDC 
grants of participants and comparison group members, os^ would 
estimate ordinary least squares regression ^uations (sef^rately for adult 
men ami won^ ) with preprogram earnings and AFDC gnmts as ctepend- 
ent variables. Multiple regression is a teditiique that estimates tte 
imlependent influeiK^ of each char£u:teristic on a i:^icular dependent 
variaMe, controlling for the influence of all other characteristics in the 
equation. For example, different^ in earnings among individuals may 
result from differemres in education and other personal characteristics, 
such as age or race, as well as differences in local unemployiiKtnt 
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condititMis. The regression technique contruls for the influence on 
earnings of local unemployment conditions and other personal charac- 
lerislics. and estimates the independent influence of all tl^se factors as 
well as program ^tiviiies cm eamin^es. All standard software analysis 
piKrkages include multiple regression programs capable of handling the 
analysis tasks described in this section. 

For a given set of outcome measures, the principal task in specifying 
tlH? regression equations to be estimated is making decisions about which 
variables to include in a given model. Because iIk objective of the 
analysis is to iitentify wlwther there arc significant differences in the 
preprtvjjam earnings and AFTXT grants of the two groups after control- 
ling for measured charwrteristics, there arc aexeral guidelines that can be 
used in making decisions concerning the independent variables to be 
included in the models. 

First, include in the model all personal characteristics of the individu- 
als at enrollment or registration who were examined as part of the 
analysis conducted for Criterion 1 above, such as age, race. educaticM). 
occupation, and handicapped status. An exception will be those who 
must be omitted because too few cases exhibit that characteristic, or those 
who must serve as the "left-out category," For example, it is likely thai 
in many states there will be too few of certain minorities (e.g.. Native 
Americans) to include them as separate variables in the model. As a 
result, one may need to collapse the five race or ethnicity group variables 
into three variables, i.e.. dummy variables for white, black, and other race 
or ethnicity status. 

Note also that in estimating the regression model, one of the race 
dummies must be omitted to serve as the "left-out category" (reference 
category) for comparison purposes. If the dummy variable for white 
status is omitted from the equation, then the coefTicienls of the other two 
dummy variables would represent the effect of being in thai particular 
group, relative to being while, on the depenctent variable. For every set 
of dummy variables ir- <uded in the regression model to capture the 
effect of a cenain chaiacteristic. one of the variables must be omitted to 
serve as the reference category for comparison purposes. Because the 
effects of the included variables arc all measured rdative to the left-out 
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caie^>ry, the results have the identical interpretation no matter which 
variable is chosen to serve as the omitted category. 

SeccHid, it is important to include previous preprogram measures of 
quarterly earnings and AFDC grants variables in the model. Piiat is, in 
examining the comparability of eamings in the immediate preprogram 
quarter, ota should inclmic i.uanerly eamings and AFDC grants from the 
second through the twelfth pixprogram quarters, given data availability. 
If. however, one were examining the comparabiMt j of eamings and 
AFDC grants in the immediate preprogram year, tlwn the second, third, 
and fourth preprogram quarterly eamings and AFDC grants variables 
would have to be ominal from the regression equation. Such variables 
are, by defmition, of the dependent variable in this case and, as such, 
cannm imiependently afTcct its value. 

TTiird, variables that are "jointly determiTOd" with preprogram earn- 
ings and AFDC should be excluded from the model. Specifically, 
exclude the dummy variables for AFDC recipient status. UI recipient 
status, and Ul benefit payments in the immediate preprogram quarter 
from all regression equations estimated over a preprogram period. These 
variables are essentially otter measures of low-income status in the same 
period aiKi cannot indepenttently affect eamings and A '^TXT grants in the 
siune period. 

A final guideline is to define tte variables used acceding to tl« 
appropriate time period. For example, if the dependent variable is 
eamings or AFDC grants in the immediate preprogram quarter (year), 
iten the unemployment rate in tte local area should similarly be defined 
as the rate for the immediate preprogram quarter (year). 

By following these guicklines. one can identify a set of indepemieni 
variables from those specified, using the procedures described ii. the data 
processing section of this chapter. These variables should be included in 
both tlw ]»eprc^ram eamings and preprogram AHX! grants equations. 
The tndepenckrt variables to inclucte in a regression model to examine 
Ite similarity of participants arel comfmriscm group nv»mbers on eam- 
ings (and AFIX* grants) in the immediate preprogram quarter are listed 
in table 2.9. Note that the interacti(His between the white anJ the age and 
education variables are optional and need no! be included in the final 
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modeL However, tl^y are included in this list to emphasize the 
importance of eonirolling for all measured dilTercnces between partict- 
jwiis and comjmriscHi group membcus. 

For each prepmgram period of interest, oik would estimate four 
regression equations that include this set of tndepemient variables. That 
is. separate models would be estimated for adult men and women and for 
both of the key inilcome measunrs. earnings and AFDC grants. The test 
for diffcrcmrcs in earnings and AFDC t;ranLs between the jmrticipani and 
eomi^son groups in the immi^iate prepmgram quarter would be based 
on a t-tesf of the estimated coefficient of the JTPA participant dummy 
variable. On a more intuitive IcveK because participation in JTPA during 
a given period cannot have an effect on earnings or AFDC grants in 
jmrvious time periods, the cocfiicient of the JTPA dummy variable in 
each of the regression mixkls described above should not be statistically 
significant (i.e., should not be signilicantly different from zero). The 
extent lo which the estimated coefficients arc statistically significant and 
deviate from zero provides evittence on the direction and magnitude of 
the likely selection bias. 

With earnings in the prcpn>gram period as the dependent variable, 
statistically significant negative (positive) cc^fficients on l!^ JTPA 
dummy would indicate thai participants were less (more) advantaged 
than comparison group members in that period cm unmeasured charac- 
teristics. If this persisted through the postprognmi period it would 
probably result in understating (overstating) the net impact of JTPA on 
earnings. Thus, if this analysis indicated that after adjusting for difTcr- 
ences in measured characteristics the preprogram earnings of JTPA 
participants were $200 less (more) than the eiunings of the comjwison 
group, then one might con.si(kr adding (suNracting) $200 to (frrni) the 
net impact estimate to adjust for difieremres in unmeasured characteris- 
tics. Nc«e» however, that because preprogram earnings and AFDC gmnts 
will be inducted as independent variables in the net impact model, the 
extent of this bias should be less in the poslprogram period. As such, 
adjusting the net imj^ct estimate for the total difference in preprognun 
earnings is likely to overeompensate for ll^ bias due to differences in 
unmeasured characteristics. 
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In analyzing the prcprogr?»m similariiy of earnings aiKi AFIX? grants 
between the two grtnips, cmc can exaniii^ r^veral different time periods. 
Fot the most part* one nhuuld be primarily intereMed in examining the 
immedtaie prep(t)gmm quarter or year and separate rcgrcssion equa- 
lions, like tiK one describe above, could be estimated for both periods,^ 
In addition* one could also estimate a regression equatiiHi like the one 
described above for each prepit>gi^m quarter ami iterive a set of esti- 
mated coeffic cms of the JTPA particif^t dummy. To the extent '»hat 
including add tionai lagg4:d values of earnings and AFDC grants in the 
equation serves to reduce the differences between tte two groups, the 
coefficients of the JTPA dummy variable should be largest (in absolute 
value) in the eariy preprogram peritxis ;md tend toward zero as the 
preprogram outc^^me is measured closer to the date of enrollment. 

The third criterion (Criterion 3) used to judge the adequacy of tte 
comparison groups is the similarity of the relationships between earnings 
(and AHXT giants) and individual characteristics for JTPA participants 
and comparison group members in the prepmgram period. This crite- 
nm, which is considerably stricter thim the previous two, is quite 
important hrcause. if the same model is generating earnings (or AFIX! 
grants) in tte two groups, it suggests that program impacts will be less 
sensitive to other piMential statistical problems. This would provide 
additional confidence in our ability to obtain unbiased estimates of 
program impacts. 

To test for dr^^fcrences in thr preprogram earnings (or AFDC grants) 
equaticms of partici|Kmts and comparison group members, one would 
estimate a modified versicMi of the regression equation described above 
to provicte infomiation on Criterion 2. Specifically, t>w would estimate 
an ^u^don that iiKluded all of the explanatory variables listed above. 
pius e tch of the variables (except the JTPA participation dummy 
variable) multiplied by the JTPA participatior dummy variable. The 
formal test of whether t!» earnings and AFDC grants ^uations in the 
preprogram period are different for jxinicipanis and c*omparison group 
members is sometimes referred to as a Chow tes! and is l^tsed on an 
test of tte hypothesis that the coefficients of the interaction terms (i.e.. 
iIk JTPA participant dummy multiplied by each of the other variables in 
the model) arc all zero.*^^ 
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Tls; three criteria and related analyses de«:ribed provide considerable 
informatirai regarding the adequacy of the ctmiparistMi gio'-jw in the 
preprogram period and the probable biases that must be dealt with. It 
Khould be emphasi;^ again ih.U these criteria arc relatively strict tests of 
the comparability of the two groups, and one should not generally expect 
noncxperimcntally derived comparison groups to meet all of them. If the 
conditions are generally satisfied, then the chances of obtaining unbiased 
program net impact estimates using standaid statistical models arc 
improved. If the criteria are sirtmgly reacted (e.g.. F-statistics of ! 0 or 
20 when approximately 1.5 is sufficient for rejection), then am should 
be very cautious in proceeding to estimate net impacts with these 
comparison groups. Instead, one should first double-check to be sure that 
the daia processing and analysis guidelines described earlier were 
followed. If the criteria are still strongly rejected, one should then 
consider obtaining assistance from a researcher familiar with these 
issues. If. as is most likely, the results are somewhere in between (i.e., 
prepn^gram differences between the two groups that are sometimes 
statistically significant, but not exceptionally lai^e), then oi« will need 
to understand the implications of these differences for interpreting sa\d 
adjusting the net impacts results. 

Estimating the Averafie Net impacts of JTPA Profirams 
The four general postprogram outcome measures for the JTPA evalu- 
ation arc earnings, whether employed, AFDC grants, and whether an 
AFDC recipient. We will discuss the specific postprogram periods for 
which the.sc outcomes will be measured for different samples of enrol- 
lees, and describe tl^ regression model to estimate average net impacts, 
A subsequent sectit»i will describe how to olnain sepai^e estimates of 
net impacts for major dcmogmphic groups, by program activity and by 
length of program participation. 

Choice of Postprogram Periods and 
Implications for Potential Additional Sample Exclusions 
The choice of the postprogram periods to be examined depends on the 
distribution of length of stay in JTPA. For example, if no individuals 
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p^cipated in JTPA longer than six months, tten for a given quarterly 
:;ample of enrollees (e.g., those who enrolled during finit quarter of PY 
1985), all m:h individuals would have terminated from the program by 
the end of the third quarter of PY 1985. As such, clings and AFDC 
grants received during the fourth quarter of PY 1985 would be the 
outcome measures for tte first complete posiprogram quarter for these 
enrollees. If, however, ttere is ct^nsiifcraWe diversity in program length 
of stay and some individuals remain in the program much longer, one 
would have to decide wl^ther to postpone tte analysis aiKi wait until all 
cases have terminate, or exclude such cases from tte analysis samples. 
Although it is genct^ly not desirable to restrict tfie participant sample to 
those who have terminated from JTPA by a pailicular date (because 
terminees could differ systematically faim nontenninees, which could 
result in additional selection biases), in most cases it will simply not be 
possible to wait for ail participants to terminate from the program and still 
provide tinwly net impact results. 

To provide timely resuhs, it may be necessary for states to chiwse a 
cutoff date that defines the program period. Any participants who are in 
the program after that point would be excluded from the analysis.'^ In 
general, we expect that defining the cutoff date to be six months after the 
end of the enrollment period for each quarterly sample (e.g., March 31, 
1 986, for those who enrolled in JTPA during the first quarter of PY 1 985 ) 
should be adequate to mwt most stales* needs. This allows a length of 
stay that is no less than six months for any individual and up to nine 
months for individuals who enrolled very early in a particular quarter. 
We expect that such cutoff dates, applied uniformly to participants in all 
four quarters of the program year, would result in excluding no more than 
10 percent of tte particii^t sample in most states. This is unlikely to 
significantly bias the average net impacts of JTPA, and should not 
significantly reduce the precision of the estimated impacts. - ' 

States that operate JTPA programs that lend to have very long program 
lengths of stay should consider extending the cutoff date to estimate 
earnings impacts for the same number of postprogram quarters. On the 
otl^r hand^ in states where JTPA services arc relatively brief on average, 
it may be possible to define a cutoff date that allows for a shorter program 
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period and* as a result net imf^cfs can be estimaied over a longer 
program period without delaying the analysis. 

As tfcscribed in the conceptual frameworic, we recommend that the net 
impacts of JTPA for adult men arri women be estimated on each of the 
four general outcome measures for a ihiee-mcmth* six-month, and 12- 
month postprogram period. Based on the data collection plan and the 
strategy to be used to rxclucte IcMig-term participants (if necessary), tin? 
research design enables one to estimate the net imp^^ts of JTPA on tiK^se 
fcnir meaMiires over a three-month pt^tprc^ram period for JTPA enrol- 
lees from all four quarterly samples within approximately a two-year 
prc^ram analysis cycle. It enables or^ to provide net impact estimates 
on these outcomes measured over a six-month postproiram period for 
the first three quanerly enrollment samples. The net impacts for a 12- 
month postprogram period can only be estimated for participants who 
enrolled in the first quarter of the program year/ ^Because of the impor- 
tance of longer-term impacts in making Judgments concerning the cffec- 
liver^ss of employment sind training programs, some states might 
consider collecting additional qu^ulers of postpnogram information for 
all individuals for subsequent analysis, ^d particulariy for those who 
enrolled in the last three quarten; oi tJ^ program year. 

Autore^ressive net inifHict mmiels. To estimate the net impacts of 
JTPA for adult men and women, we recommend that an auloregre^isivc 
model be used. Using this approach, ordinary least squares regression 
equations woula be estimated for each of the 12 outcome variables, that 
is, four outcome measures in each of three different postprogram periods, 
separately for adult men and women. The autoregrcssive ap}»x>ach is so 
named because preprogram values of the ouic-ome measures — quarteriy 
earnings and AFDC grants — are also included as imkpemient variables. 
This af^roach has the primary advantage of controlling for any differ- 
ences in measured char^teristics between the two groups that remain 
aftei the matched comparison groups are selected, which Iwlps to 
minimize the problem of selection bias. 

To control for potential differences in the channrteristics of partici- 
pants and comparison grcnip members to the extent possible, it is 
recommencted that r net impacf regression mcxiel be a slightly ex- 
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f^ded version of the models used to determine whether the comparison 
groups meet luc preprogram cwnparabilily Criteria 2 and 3. The only 
changes in the indepencknt variables to be included in the basic i^t 
impact model, as compared to the variables included in the preprogram 
models discussed above and listed in ta Me 2.9. arc as follows: 

Tibi»2J 

SamplB liM^ieM»l VirteMwi to ii%!iHl8 ta M(rf8l te 
Exsmba Aifeq^isy of Compaitem 6roo|» 



Age 

Age^uared 

Black dummy 

OSwt non-white dummy 

Hi^^schooi graduate dummy 

PvlsX high-school 8ducaik)n dummy 

(As») X {while dummy)— opttoiwl 

(Age square]) x (white dummy^-optmnal 

(High-schooi gr^i^e dummy) x {\i^lte dummy)-H)ptiof^) 

(Pt^ h^h-si*ool «lucatlon dummy) x (white dummy)— opticmal 

VetefBJi dummy— for m^ only 

Handicapped dummy'-if n^sured ccmj^raWy for l«j!h groups 

Recmt EmploymoM Eq»^nm 

Set ot eight of^igit DOT dumm^: for example, altoi^ng professional to t>e the left-out category « 
the eight occupation dummies would correspond to clerical and saia*: service; agricultural, fishery, 
and fo^ry: proc^ng; machine trades; tochwork; structural work; and mrscet^r^i^ 

f^nGyMi)0/2m quarterly ^mings--separate variable for preprogram quarten^two through 12, data 
pennittjng 

Preprogram quarterly AFDC grants— separate vana!>les for prepri^ram qiarters two through 12, 
data penrntting 

Later NirMOato 

Ur^mptoyment rate during the immediate preprogram qi^rter 
Urban location dummy 

Prograffl Psr^jntimi Varlaiiin 

JTPA partfclpant dumnty (or alternatively separate dummy variables for prt^ram activities) 
OttitrVariibJts 

Set of dumn^es for the quarter of enrollment or regtstratKm: for t nstam^e. allowing the first r] t*arter 
to serve as the ieft-oui cate^ry. dummy variables for whetter participants (conH>arison r.roup 
memt»rs) enrolfed (registered) in qtarter tm, three, or tour 
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1 . Quarterly comings and AFDC grants in the immediate preprogram 
qumter iihould be included in the net impact model. 

2. TTie net impact model should also include the UI recipient dummy 
variable. UI benefits earned in the immediate preprogram quarter, 
and IJk; AHX! recipient dummy variable. 

3. The unemployment rate should be defined according to the postpro- 
gram period for which the model is being estimated. 

Thus, following these guidelines, one can estimate autorcgressive 
models separately for adult men and women, and the estimated coefTi- 
cienl of the JTPA jMulicij^t dummy variable reprwenis the average net 
im{»ct of JTPA on earnings and AFDC grants for the three pc^tprogram 
periods of interest (i.e., \br^, six, and 12 months). For dependent 
variables expressed in dollar terms — earnings and AFDC grants— the 
coefficient of the JTPA i^icipanl dummy variable can be interpreted as 
the average dollar impact on a given outcome measure. Dividing the 
estimated dollar impact by the mean earnings or AFDC grants of 
comparison group members results in an estimate of the percentage 
change in earnings or AFDC grants due to JTPA. 

For dummy depen^t variables (i.e., whether employed in a particu- 
lar period, or whether receiving AFDC grants during a particular period), 
the autoregressive net impact model is equivalent to a linear probability 
model. The model essentially estimates the effects of various factors on 
' lie probability of a certain event occurring, for example, having positive 
earnings in a given posiprogram period. As such, the estimated coeffi- 
cient of the JTPA participant dummy variable can be interpreted as the 
average percentage point change in the prol^bility of working or 
receiving AFDC grants due to JTPA. By dividing the estimated percent- 
age point change by the mean proporticm of comparison group members, 
one can obtain an estimate of the percentage change in the probability of 
working (or receiving AFDC) due to JTPA. 

Obtaining Net Impact Estimates for Various Subgroups 
The mcxiels described above focus on providing overall estimates of 
the net impacts of JTPA for adult men and women. Determining whether 
JTPA effectiveness varies by the type of program activity and by 
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persona] characteristics has import^t fK>iicy and planning implica- 
tions.^ Because JTPA program activities and particip^ chanK:teristics 
can change cc»isicteraWy over time, knowledge of how program net 
imqiacts vary among them would help interpret time trends in JTPA's 
impacts* Furthermore, information on which program activities and 
services work best for given types of participants can fm^vick valuable 
information for targeting future employment and training programs* 
Although the apprrach to estimating i^t impacts for different groups is 
formally identical, wilder tin; giuup refers to the type of program 
activity or to individual cham:teristics, ^itional selection bias is likely 
to arise. In the r^xt s^tion we describe how to modify the autoregressive 
earnings and AFDC grants mocfels to estimate the mrt impacts of JTPA 
for various groups, and review Uk^ additional biases that one must be 
aware of in interpreting the results. 

Net impacts by participant characteristics. In gei^ral, specific group 
effects are estimated by including in Urn regression equation an interac- 
tion term that represents the product of the dummy variable for JTPA 
participation with the variable for the group of interest. Suppose oro is 
interested in testing whetl^r the net imfmct varies by a characteristic that 
is represented by tlw thrre dummy variables Z^, and One might 
think of the three variables as representing lac^ or ethnicity categories 
(white, black, other}.^^ Then, the only modificaticm required to the 
autoregressive model ctescrib^ above involves replacing the JTPA 
I^icipation dummy variable with three variables that each involves the 
JTPA dummy variable multiplied by tMie of tte thrte variables represent- 
ing the particular group (i.e., JTPA Z^, JTPA x Z^, JTPA x Z,). The 
coefficients of these three variables are estimates of the net impact for the 
three groups of interest.^ 

In attempting to disaggregate JTPA net imp^ts across groups, it is 
important that the group characteristics ^Iso be include in lbs model as 
control variables to account for differences in the general level of 
earnings (or AFDC grants) across these groups. In our illustraticm, the 
three Z^ variables must be in the mcxktl separately so that estimated 
net impacc coefficients only capture outcome differences due to JTPA 
i^ross these groups and do not also capture tne average diiferenc^s in 
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ouicoim^s due to the ttwirii^clves. In addiiion. it is also imporvani that 
tlK: groups be mutually exclusive ^uid exhaustive. 

For example, suppose the Z 's refer to various participant age catego- 
ries: 22-34. 35-54, and 55-64. Then two types of problems can arise in 
estimating the net imf^ts for these age groups: 

1 . Receding errors can occur in creating the variables (e.g.. ranges of 
22-44. 35-54. 55-64) that result in overiapping the age ranges so that 
individuals age 35-44 would appear in both of the first two groups 
(i.e., the groups are not mutually exclusive). 

2. Individuals in the sample may not fall into any of the three age 
categories created (i.e.. the groups arc not exhaustive). 

This could occur if some j^rticipants were younger .lian 22, older than 
64, or if there were a gap in the age ranges used. If the groups arc not 
exhaustive, then all of the fmrticipant observations that do not fall into 
one of the categories would be treated as compari.son group members, 
which would rcsuli in biased estimates of the net impacts of JTPA for the 
other groups. 

At a minimum, we recommend that states examine differential im- 
pacts by race, education, age. UI claimant status, AFDC recipient status, 
and preprogram earnings for those individuals who had preprogram 
eamings. Becau.se individuals' preprogram characteristics cannot be 
affected by JTPA. no additional selectivity bias is introduced in disaggre- 
gating JTPA net impacts by demographic groups,^^ However, this is not 
likely to be the case when examining whether JTPA effectiveness varies 
by prt^ram activity. 

Net impacts by program at'tivity or ser\'k'e. In principle, to probe 
beneath the average net impacts of JTPA and provide information on the 
program activities that contributed to the average effects, one would 
perform an identical interaction analysis to the one described previously, 
using Z,, Z,, Z, and to represent classroom training, on-the-job 
training, job search assistance, and other program ajwiistance respec- 
tively. Then, if c represents the estimated coefficient for the interaction 
term between the JTPA dummy and Z, then c, is the estimate of the 
average net impact for CT, c, would represent the estimated net imj^ct 
for OJT, c, would represent the estimated net impact for JSA, and c^ 
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would represent the estimated net imp^t for other JTPA mrtivities. 
There is, however, potential selection bias that can threaten the internal 
validity of tte by-program activity net impact analysis. 

SiKrh bias relates to tt^ nomancbm ^signment of JTPA participants 
to program activities. As described above, the assignment of program 
activity is likely to be based on the agency's perception of an individual 's 
ne^^ and abilities. To tix extent that this assignment pnxress is l^sed 
solely on the measured characteristics of particip^ts, such as ^e, race, 
sex, education, and preprogram earnings, this will not bias tlK net 
impels by prc^ram activity, as these characteristics will be inclwJed in 
the net imp^t model. But if the assignment of program activities is Imsed 
on immeasurcd char^neristics, such as motivation aiKl ability, and those 
unmeastired characteristics also affect earnings, then selection bias 
results. Thus, one must he very careful in interpreting i^t impacts by 
pn^ram miivity. 

Obtaining Net Impact Estimates hy Program Length of Stay 
To investigate whether the net impacts of JTPA vary by length of stay 
in the program, one would estimate an autoregressive model like those 
described eariier, with the overall program participation dummy variable 
replaced by a JTPA variable that measures length of program stay in 
terms of total w^ks or, more appropriately, total hcHirs. If the effects of 
length of stay on the outcomes are approximately lii^ar, a ccmvenient 
specification involves a model with a JTPA participation dummy and the 
participation dummy internet^ (i.e., multiplied) with total weeks (total 
hours) in the program minus average number of weeks (t(«al hcmrs) in the 
program. In this specification, the coefficient of tte JTPA dummy 
represents the estimated impact of JTPA at the average length of stay 
(average total hours), and tl^ coetticient of the interacticm term is an 
estimate of tl^ dollar impact of an additicMial week (hour) of program 
participation. 

Although the autoregressive earnings mode! controls for differences 
in measure characteristics between short- and long-term participants, it 
is likely that some differences in unmeasured characteristics remain. 
Individuals who leave the progiam early may be less motivated or. 
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alternatively, may have found employment on iheir own. On the other 
hand, ii di viduals who stay in tlw program a long time may do so because 
they have fewer employment opportunities. Length of stay is also likely 
to depend cm the type of program activity and SDA characteristics. 
Because of these additional selection bias problems, caution is needed in 
asserting a causal relationship between services and program impacts by 
length of stay. 

Adjustments for Potential Data and Desiffn Deficiencies 

In addition to the problem of poteni^al selection bias, there are some 
deficiencies in the UI earnings, JTPA , and ES data that may affect results. 
UI Wage Records are inct)mplete. They do not reflect earnings from jobs 
that are not in covered employment, or earnings from jobs located across 
the bonier in other states. The JTPA and ES data are deficient because 
there is inadwjuate information on whether ES registrants p^icipaied in 
JTPA, which may result in a contaminated comparison group. In this 
section, we briefly discuss the likely extent to which the basic impact 
estimates will be affected by these data and design deficiencies and the 
types of adjustments that may be iKcessary. 

In the earlier data processing discussion, procedures were outlined 
that could reliably exclude those individuals from the comparison grcnip 
who were currently participating in JTPA, who had participated in JTPA 
in the previous year, or who participated during the postprogram periods 
being examined. If it is not possible to implement these procedure, the 
comparison group will be contaminated to a certain extent. Such 
contamination would lead to an underestimate of the net imfiracts of 
JTPA, since it would dilute ibe treatment, as some comparison group 
members would have also received JTPA activities and ^rvices. 

Although the ES is one source of applicants for the JTPA program, and 
one might expect that contamination could be high, exist ing data indicate 
otherwise. For example, tesed on data for the State of Washingtcm for 
PY 1985,only0.1 percent of all ES registrants active during the year were 
recorded as having enrolled in JTPA programs. Only 0.3 percent of those 
who were economically disadvantaged enrolled in JTPA. Although the 
figures are somewhat higher for enrollment in any training activity (e.g.. 
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JTPA» Job Corps, WIN, other)— 1.0 percent for all applicants and 3.1 
percciii for those ectmomically disadvantaged — even these participation 
rates ate small enough to be safely ignored. 

In states that have higher protmbilities of economically disadvantaged 
ES registrants enrolling in JTPA, and in which it is impossible to exclude 
those who participate in JTPA from the comparison group before 
conducting the net impact an^ysis, it may be n^essary to make some 
agpiegale ^justment to the net impact estimates. Specificsdly, if p (q) 
is an estimate of ttw proportion of the adult men (women) in the 
comparison group participating in JTPA during the period of enrollment 
through tlw postprc^r^ period (i.e., from oi^ to two years), the 
estimated average jwogram net imi^cts for adult men (women) should be 
multiplied by l/(l-p) (or l/(l-q)) to adjust for this problem. 

The second major data deficiency is that UI Wage Records do not 
include jobs in uncovered employment, or earnings fnmt jobs in other 
states. However, the omission of earnings due to these problems biases 
tte estimated impact of JTPA only if program jwrticipation cau.sally 
affects tte probability of working in uncovered employment or the 
likelihood of woiiiing in another state. Given the focus of JTPA on 
employment in the private sector, this should be less of a problem for ite 
state-level net imi^ict modei. Also, in order to create a meaningful 
adjustn»nt. one wcnild need informatimi on interview-reported earnings 
and UI earnings for both groups in the postprogram period, which will 
not generally be available. Thus, the best oi% can do is acknowlajge tte 
potential problem and indicate that the net impact estimates are Imed on 
the reasonable assumption that JTPA docs not affect the probability of 
woricing in uncovered employment or working across the bonier in other 
states. 

Cost Analy^ and Benefit-Cost ComparfaM>ns 

The estimated net impiK;ts of JTPA program activities on particij^ts* 
pfwtprc^ram labor market experiences can be used to estimate the 
benefits of the JTPA forprogran participants and. under certain assump- 
tiwis, the benefits of the JTTA to society as a whole. To determine 
whetter the JTPA is an effective use of public resources, however, one 
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must assess the cofAH of providing JTPA mrlivitics and comj^re iIk cosIh 
to tte bemrfils of the pit^ram. U is a serious conceptual error to assess 
a social program on ti^r t^sis of overall costs or benefits <doiK* Moreover, 
to make informed decisions about the design of the program, poli- 
cymakers must know both the costs and benefits of specific program 
activities. TTiat is. program activities that yield relatively small benefits 
may yet be very effective when compsu^ to the costs involved, since 
what matters is the siKial rate of return on the dollars invested in each 
participant, just as it is tlKr rate of return on capital that matters for any 
private sector investment. 

Wl^n all costs and benefits are accounted for, u ber^fiKost analysis 
judges the sM ial efficiency of a prognun. It determit^s whether the 
value of the goods and services available to society and by extension, to 
the members of society are greater as a result of the program. To make 
this determination, the benefits aa- typically assigr^ a monetary value* 
and their present value is compared to the present value of tl^ moi^tized 
program costs. Assuming that all present and future benefits and costs 
are identified, appropriate monetary values arc assigned and an appropri- 
ate interest rate is used to discount future benefits and costs to their 
present values. JTPA could be ccmsiden^ a worthwhile use of fHiblic 
resources if ( 1) the present value of the benefits is larger than the present 
value of the costs, or C2 ) the rate of retum, that discounts the sum of costs 
and benefits to zero, exceeds the socially specified rate of return. 

Although tiK process of ccmducting a benefit-cost analysis is straight- 
forward, there is a variety of issues that limit the validity of such an 
analysis. Given data limitations and other issues* it will generally be 
possible for states to conduct a comprehensive benefit-cost ;»nalysis to 
provide a defmitive estimate of JTPA*s social retum on the investment. 
Nevertheless, the general approach described below is useful in organ- 
izing information on benef ^ts mid costs, and enables states to c^tain some 
sense of the effectiveness of the JTPA and the conditions under which 
JTPA can be regarded an efficient use of public resources. 

The discussion begins with a brief description of a benefit-cost 
framework for analyzing the ctTeciiveness of the JTPA. TTien we briefiy 
describe the use of the net impact estimates to measure some of the 
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imponant nocial bencnis due to the JTPA. Wc subM.'qucnUy describe 
how to conduct a cost unaly*sis lo c)»ttniate the mai^snal cost of Mrrving 
^itional JTPA particifKUits and iIk? marginal costs of dif fereni jirogram 
activities, A discussion of the issue of discounting future benefits and 
costs so that cimiparisons con be made in present value temis follows. 
We comrlude by considering a few additional comparisons thai should be 
made to detemiine how sensitive the overall conclusions an? to cenain 
assumptions. 

Benefit-Cost Framework 

The benefit-cost framework presented in table 2.10 lists the major 
benefits and cmts that would ideally be accounted for in conducting a 
comprehensive benefit-cost analysis. As an aid for keeping track of the 
different benefits and costs, they arc presented from thn^e perspectives, 
the j^rticipani^ the taxpayer* and society as a whole. The first class of 
benefits and costs coasists of those benefits a*ceived by. or costs home 
by, piugram participants. The participant j^rspeclive is important be- 
cause it dieds light on an individual's incentives and willingness to 
participate in the program without coercion. The taxpayer inrrspcctive. 
sometimes referred to as the nonpanicipant perspective, is important 
because it reflects the effects of the program on the government budget 
ami t}^ willingness of taxpayers to support the program. 

The most inclusive set of program benefits and costs arc those 
^cniing to society as a whole. These are simply the sum of benef ^ts and 
c^osts received, or borne by participants and taxpayers (that is. all 
members of society), taken .separately. These represent a full accounting 
of alt costs and benefits involved in operating the program. It is important 
to note that tte social i^rrspectivc ignores transfer j^ymenis between 
segments of society, that is, between participants and taxpayers, and 
examines instead whether the program results in a net increase in the 
resources available to society.^** This is the appropriate perspective for 
a governmental body to take in examining the overall effectiveness of the 
JTPA. 
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TiWBMI 



• increased output 

" Postpn^ram output ♦ 

- Program output 0 

• Rmluced receipt ot income trans'm 

- Reduced «wltare fwyments, .egardiess 
of whether still on i^ltan^ 

- Increase tax payments 

• Reduced use ot atterr^tive social 

programs 0 

• Nonmonetary Imi^rts 
' Reduced crime 

* Impro^ v^rtt attitudes ot 
participants ♦ 

- Improved mental and phys/'^al health ^ 



0 
0 



Costi 



• Pribram operating costs 
(e.g.. rent, staff w^es. and 
fringt^, materials and suppli^, 

and overhead admintstrative costs) 0 

» Participant opportunity costs 
{e.g., forgone earnings or home 

prcKluctton) - o 

• Tranters to participants 

(e.g., stipends) ♦ 0 

• Costs of participation 
<e.g„ transportat«)n. child 

care, extra clothing, and food) ^ 0 ♦ 

• Psychic costs 

(eg , stress of studyino 
and beifHj tested, separation 

from children) - 0 
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Program Benefits 

The major benefit of the JTPA from the social perspective is the 
increase in output produced by particip^tH. Concefrtually, two types of 
gain should be distinguished: ( I ) the increase in pcnaprugnim output, 
measured by the increase in earnings of the panicipnts, and (2) the 
inciease in output produced while an individual participates in the 
program. For the most part, the current-program (as opposed to pre- or 
postprogram) output due to the JTPA is likely to be small, panicularly fw 
(^icipants in dassroom training and job search assistance-program 
activities. Only for OJT prc^rams is the vali^ of program output likely 
to be positive, and even for tlKse pn^jams it is difficult to assign 
apprrpriate monetary values. Because of the difficulties involved in 
measuring the value of prc^ram output, as well as in mea.><uring the value 
of crthcr nonmoTJlary benefits, such as reduced crime or improvwi 
mental or physics 1 l^slth of participants and their families, we recom- 
mend that states 'lo wA attempt to directly measure these benefits, but 
recognize their potential importance whe» iiscu.^sing the overall results 
from the benefit-cost analysis. The primary benefit to be measunwl, 
therefore, is the increased po.^tprograiu output due to JTPA. 

The participant-comparison group differences in earnings in liK 
postprogram period are used to measure the increase in output of goods 
and services available to society during that period due to JTPA. This is 
a reasonable jnocedure provided that JTPA participants do not find jobs 
in ihe postprogram period at tte expcnj^ of mher disadvantaged per- 
sons.'' It is beyond the scope of the state-level model to assess the extent 
of such job displacement. As a result, tb^ benef ^t-cosl analysis is limited 
to determining whether the social benefits from receiving JTPA activi- 
ties arc greater than the costs to society of providing thcKic activities, as 
measured by the change in total posiprograni earnings due to JTPA 
activities. 

Two issuer arise in translating participant earnings gains into a 
measure of the increase in output of goods and services available to 
society. Finsi, one must determine how to extrapolate the jx>siorogram 
gains observed for the periods from three numlhs to one year following 
termination, into subsequent periods. For o amplc, if the three-month. 
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six-month, and 12-monih net emuings impacts imply similar gain^i per 
quarten then it may he reasonable to assume that the gains persist over 
time. However, based on previous studies, it is likely that t!^ gains 
decline over time, and information through just the first pwtprogram 
year may not be sufTicient to estimate teliable time trends for t}^ purpose 
of extrapolating future gains. This cmphusiEeK the tmportano^ of aug- 
menting the basic design with the collection of aHditionai ]^pn;>gram 
data to more precisely estimate the long-term gai*^ . . tte JTPA, If this 
is not possible* at a minimum the bci^fit-cost analysis should imiicate 
whether earnings gains observed during the oiw-yesur follow-up period 
aie sufficient to make tl^ program worthwhile, ami if not« indicate 
whether JTPA would be viewed as a worthwhile social investment if the 
gains persisted at their currrm level for up to five years,^^ 

A second issue cont^ems adjustments that should be m^ to earnings 
gains to account for fringe benefits. That is, if the increase in output is 
equal to iIk increase m compensation paid to those who panicipaie in 
JTPA, then although this compensation is primarily in the form of 
monetary earnings* adjustments for nonmonetary earnings should also 
be made. Fringe benefits ir-'Iude pensions, l^alth and other forms of 
insurance, and payments on behalf of iIk* worker for unemployment 
insurance, workers' ci>mpensation, and FICA. As a rough approxima- 
tion^ insurance and pensicm benefits for workers served by JTPA are 
estimated to be approximately 10 percent of monetary benefits {Wood- 
bury 1 9X0), and payments to government programs are approximately 10 
percent as well. Thus, wc recommeml that the net earnings gains be 
multiplied by 1 .2 to adjust for fringe l:^nefits in deriving a measure of the 
social benefits due to JTPA. 

To summarize, the benefits to be measured and included in the benefit- 
cost analysis include only tl» increase in postprogram output due to 
JTPA. This is approximated by the imrreasc in postprogram before-lax 
earnings, as measured by the estimated coefficient of \Ik JTPA dummy 
variable in the net imj^ci equation — adjusted for potential data and 
design deficiencies as ctescril^ in the previous section — and subse- 
quently adjusted for fringe benefits. Procedures will be developed to 
ctetermine how the increase in earnings over the first year should be 
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extrapolated to yield estimates of increases in postpmgram output in 
subsequent years. These steps will yield an estimated stream of future 
benefits for both aduh men and women. Individual vaiues in the^: benefit 
streams will then be weighted by the proportion of men and women 
served by the JTPA in the particular program yean to generate an 
estimate of the aggregate benefit stream due to the JTPA, 

JTPA Costs 

As iiKiicated in table 2. 1 0, therc arc several dilTcrent cost components 
in a benefit-cost analysis. The major cost categories include ( 1 ) program 
operating costs, (2) participant opportunity costs, (3) transfers to [^rtici- 
pants, (4) costs associated with participating in the program activities* 
and (5) psychic costs lo participants of j^dicipating in the JTPA. 

The major costs from the social perspective arc the program operating 
costs. These include direct operating costs, such as i^nt, salaries for 
instructors, and costs of materials and supplies, and indirect or overhead 
costs, such as thi>se involved in managing and administering the program 
overall. Because no fee is charged for program participation to those 
eligible, the operating costs are not considered as costs from the perspec- 
tive of program participants. However, operating and administrative 
costs do involve the use of resources that have alternative uses- They 
represent real costs from the perspective of the taxpayer and society as 
a whole. Thus, in table 2.10, such costs are represented . s a zero to 
program participants and as a minus in the other two columns, 

A second important component of cost concerns the earnings oppor- 
tunities and home production that participants forgo while participating 
in the program. These forgone earnings and home production are clearly 
costs to participants and, to the extent that less output is produced 
because workers were participating in the JTPA, the forgone output (as 
measured by forgone earnings or the value of forgone home production) 
is a cost lo society as well. Although previous studies have recognized 
forgone earnings and lost home production to be impcmant elements of 
program cost, because of data limitations these components are almost 
always excluded from the final benefit-cost comparisons. We also 
recommend that this cost comjxjneni be formally omitted from the 



f O J 



ERLC 



1 14 Evaluating Social PiubkmK 

benefit-cost analysis. Although the social pmgram costs arc umkrstated 
by this exclusion, given the relatively brief length of participation in 
JTPA and the questionable nature of the assumption that considerable 
output was forgone when previously nonwoiting individuals partici- 
pated in the program, wc believe this is justified. In interpreting tte 
results of the benef.t-cosi comparisons, however, one should indicate lo 
what extent the overall assessment is likely to be sensitive to omitting the 
social cost of forgone earnings and lost home production.-" 

Other potentially important costs of JTPA from the taxpayers' 
perspective are tmnsfers to jwiicipanls in the form of the money value 
of classroom materials, stipends, transportation, child care, and food or 
clothing allowances. Although such costs arc much less important under 
JTPA than underCETA. they could be considerable in some cases. Note, 
however, that such transfers are a cost from the taxpayers* pcrsp«:tive. 
and a benefit to the participants that receive them. As such, transfers do 
not affect the cost-benefit analysis from the social perspective because 
the loss to the taxpayer is cancelled by the gain to the participant. 

Other potentially important costs are the direct costs participants ncur 
in participating in JTPA activities as well as any psychic costs, 'lliese 
psychic costs are inherently unmeasurable. and are included in the 
conceptual frameworic only for the pujpose of completeness. The costs 
incurred by participants in attending classes or (Kirticipating in job search 
activities require data thai must be obtained by interview from individual 
particij^ts. Brcause of the large expense involved in acquiring such 
information, and given the fact that these costs are likely to be a small 
share of the total cost of the program to the individual and to society as 
a whole, as a practical matter these costs are omitted from the final 
benefit-cost comparisons. 

To summarize, the costs of JTPA to be measured and included in the 
benefit-cost analysis will be limited to those involved in operating the 
program, that is. the sum of rent, staff, materials and supplies, and 
administrative costs. In inteipreting the benefil-cosi analysis compari- 
sons, however, it is important to recognize that many of tte social costs 
of JTPA have not been measured, and that these unmeasured costs could 
affect the overall assessment of whether or not JTPA is an efficient use 
of social resources. 
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Estimating Progrtm Costs 

Tte berrefitH from JTPA will be expressed in lerms of incremental 
dollar gain^ per tmlividual adult participant. Theiefoie. ll^ cost analysis 
must similarly animate ih^ incremental (i.e., marginal) outlay in dollar 
terms per individual adult panicipant. Many ol^tacles exist in deriving 
reliable estimates of the mai^ginal costs of serving JTPA participants: 
problems of data omission, inconsistent aggregation, difficulties in 
allocating input costs among joint outputs, and ambiguity involved in 
imputing prices of existing agency or SDA resource inputs. However, 
statistical methods that overcome several of these problems and provide 
useful information on the marginal costs of employment and training 
programs are available. 

The primary data source for the cost analysis is the JTPA * uul Sfufus 
Rep^m ( J ASR). The JASR provides for each SDA the ch .tistics of 
program terminces and information on program outcomes a,*u costs for 
Title II-A and Title 111 progrxwns funded under the JTPA^- Fortunately, 
these data are provided separately for adult and youth participants in Title 
Il-A programs. 

The JASR data have sevemi advantages. In addition to being in a 
standardized format with unambiguous definitions of all information 
items, the JASR contains data on total fcde»^ expenditures (but not total 
social costs) in operating the JTPA, as well as some infomiation on tl^ 
socioeconomic characteristics of adult progi^ terminees that can be 
used to standardize the cost analysis. The more im]X)rt^t variables are 
the number of terminees by sex, age, education, race or ethnicity group, 
welfare-recipient status, limited English language proficiency, and 
handicapped status, and the average number of weeks p^icipated. 

These participant characteristics can be thought of as inputs that enter 
the employment and training production process and have obvious 
instructional and resource implications that afTcct costs. For example, 
those participants with limited English language proficiency will proba- 
bly require mwe program resources to complete training successfully. 
As such, these characteristics can be to standardize t!^ relationship 
between total costs and participants served to obtain estimates of mar- 
ginal costs as described below. 
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The ] ASR a>niains two mapr limilations. The JASR dcxrs not contain 
( 1 ) data on ihc number of lenninecs by type of program activity, or (2) 
information on administrative and other costs incurred at the stale level 
in operating JTPA programs. Without information on the number of 
individuals served by type of program activity, one cannot identify the 
marginal costs of specific JTPA activities. This precludes comparing the 
marginal benefits and marginal costs of different program activities and 
services. Asa nrsull, one cannot identify ihoj^ activities and services that 
anr relatively most eficclivc. One can only evaluate JTPA as a whole. In 
addition, without infonnalion on the co.Hts incurred at the slate level, the 
marginal costs of serving JTPA participants as derived from JASR data 
arc understated. 

To overcome these pn)biems, we offer two recommendations. First, 
every enbrt should be made to obtain data on the number of adult 
temiinces that participated in various program activities and services 
during the program year. At a minimum, it is useful to have data on the 
following: CT-only terminees. OJT-only terminees, JSA-only ter- 
minecs, temiinces who only participated in some other activity, and 
lemiinecs who participated in muhiple activities. This information must 
be obtained from each SDA for the same period in which the tennim!e 
characteristics and program i .>sts on the JASR are nrpoiled. One could 
implement the following steps to obtain the necessary information for 
PY m9: 

1 . Ca^ate a woriting file (on tape or disk) of all persons who terminated 
from JlPA Title II-A programs in any SDA in the state during PY 
I9S9. The flic should include the person's age, data on all program 
activities and services participated in. and an SDA identifier. 

2. Kxclude from the file all persons who are 21 or younger. 

3. Create variables that represent each type of activity of interest and 
that may have diffcitrnt cost structures. For example, as indicated 
above, it is important to differentiate the costs by type of activity as 
well as costs for those who participate in only one activity vs. 
multiple activities. TTiis can be accomplished by creating five vari- 
ables, the first four of which would simply be dummy variables 
indicating whether the only activity the person participated in was 
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CT. OJT, JSA, or olhcr, and a fifth variable indicating whether the 
person participated in any combination of these activities. 

4. Ctieate a separate subfile of adult terminces for each SDA. 

5. Create counts of the number of individuals in each SDA in eiKrh of 
the five program-treatment types, and merge these counts with the 
JASR data for each SDA, 

Our second recommendation concerns how to handle costs incurred at 
the slate level in the operation of JTPA programs. Conceptually, the 
actual or imputed JTPA expense incurred at the state level should be 
ad(ted to annua! pmgram-year SDA total costs to obtain a better estimate 
of the overall social costs of JTPA. Provided information is available on 
the total costs contributed by the state to the operation of JTPA, one can 
allocate these to the various SDAs, For example, one methcKl is to 
assume that the overhead costs incurred at the stnte level in support of 
various SDAs are proportional to the number of adult terminces in each 
SDA. Thus, to allocate state-level costs in operating JTPA programs to 
ttie different SDAs, one could multiply total state costs by the ratio of the 
number of adult terminces iti a given SDA to the total number of adult 
lerminees in all SDAs. Such a procedure would, in part, overcome the 
limitation of the JASR data described above. If it is not possible to obtain 
estimates of costs contributed at the state level to the o|x?ration of local 
JTPA pmgrams. this limitation has to be recognized in interpreting the 
results of the benefit-cost analysis. 

With the basic data set and adjustments described above, one can 
estimate a program cost function that provides information on tlw 
marginal cost of ser/ing JTPA participants. Using ordinary least squares 
regrcssion techniques, one could estimate a regression equation with 
total federal expenditures plus allocated costs incurred at the state level 
(if possible) for the SDA as the de|X!ndent variable, expressed as a 
ftuKtion of the following variables: 

1. Number of adult men terminces. 

2. Number of adult women terminces. 

3. Number of adult terminces who are: 

black 
Hispanic 
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other nonwhite 

students — high schtx>l or less 

high school graduates 

aged 22-54 

welfare recipients 

single household heads with dependent children 
UI claimants 

limited English language proficiency 
handicapped 
4. Average number of weeks of participation. 

With these independent variables in the regression equation, the 
coefficient of the variable "number of aduh men temiinc^s" would 
represent the marginal cost of serving additional male adult participants 
in JTPA on average, and the coefficients of the other variables in the 
model would capture the extent to which the marginal cost varied for 
serving persons with specific characteristics. 

If the pnx-edures outlined above are followed so that data on ine 
numbers of terminces b> program activity are obtained for each SDA. 
then one would estimate a second regression equation like the one above 
except that the "number-of-adult-men-ierminees" and the "number-of- 
adult-women-terminees" variables would be replaced by the following 
four variables: the number of CT-only temiinecs, the number of OJT- 
only terminet.4, the number of JSA-only lerminees. and the number of 
terminees that participate in multiple activities. In this formulation, the 
coefficients of these four variables would repre.senl estimates of the 
marginal cost for each of the different types of program activities. If there 
arc enough observations, the numbers treated can again be separated by 
gender, forming eight categories so that costs by activity can be esti- 
mated as a function of gender. 

These e.stimates of the marginal costs of serving adult JTPA partici- 
F«nts (either overall or by program activity) are then compared to the 
marginal benefits from the program in terms of jncrcas«i postprogram 
output (eitter overall or by program activity) to state whether JTPA is an 
effective use of public resources. 

One additional potential limitation of the cost analysis should be 
noted. This concerns the i.ssue of sample size. Since the analysis is based 
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on SDA-level data* the number of oteervations available in an annual 
cross-sectioi^l analysts equals the number of SDAh in the state. In 
lelatively small slates with few SDAs, working with annual data may 
yield insufncicnt observations to estimate a cost model like the one given 
above and inhibit ability to obt£un reliable estimates of fm>gram marginal 
expenses. A solution is to pool quarterly data over a few years and include 
dummy variables for different quarters to account for wasonal cost 
differences and other lumpy costs. 

Bemfit-Cost Comparisons 

Three data elements are required to condiK:t a cost-benefit analysis: 
estimates of the benefit stream over time, estimates of program costs over 
time, and the intercrst rate used to discount future benefits and costs into 
present dollai^ In this section we indicate how to discount tl^ future 
benefit stream so that the present value of benefits can be compared to the 
current program costs, and indicate the criteria to be used to measure the 
net effectiveness of JTPA as a social investment. The discussion con- 
cludes with examples of comparisons that should be made to determine 
Ik)w .sensitive the overall conclusions are to alternative assumptions. 

Because the bei^fits of an employment and training program occur 
over time, one must translate this stream into a common reference period. 
Conventionally, this involves discounting future dollars into their pres- 
ent value, using an interest rate that approximates the alternative costs of 
the funds invested. The two interest rates that have been umi most often 
in such pnxresses are the rate of return on investment in the private 
sector — ^historically averaging around 10 percent before taxes — or the 
long-term rate of growth of the economy — historically, asDund 3 pea^nt. 
We believe that the lower rate is preferred for evaluating an investment 
in human capital from tlie point of view of society as a whole. Because 
tt^re is much disagreement abcHit which is the mor^ appropriate interest 
rate to use, however, we also recommend that states examine how 
sensitive the main results are to using a higher figure such as 1 0 percent. 

Using a 3 percent discount rate, one would measure the net effective- 
ness of JTPA by calculiUing a benefit-cost ratio, where the numerator is 
tte present vaiue of the incremental benefits due to the program (i,e., Z 
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B(l)/( 1.03)'). the denominator is the pa'scnl value of the costs incurred 
(which requires no discounting, since all ci>st.s arc incurred in the cunrenl 
period), t refers to the poslpn^gram years in which benefits are realized, 
and n is the last year in which benefiis arc rcali/cd. Within the numerous 
limitations described earlier, the JTPA would be regarded as an efficient 
use of public rcstiurces whenever the benefii-cmt ratio exceeded 1.0. 

In addition lo obtaining the main bonefil-cost results described above, 
we believe it is impt>nant that benefit-cost ratios be calculated to 
demonstrate how sensitive the conclusions are to alternative assump- 
tion*. In particular, allemaiive ratios should be calculated for the 
following: 

1. A 10 pereent discount rale. 

2, Benefit estimates that do not include adjustments for selection bia.s 
or for fxncniial contamination. .Since each set of estimates rests on 
a different set of inherently untestablc assumptions, it is important to 
know how sensitive the overall conclusions are to the size of these 
adjustments. 

A range of program benefits and costs that rcfltn:ts the fact that the 
main estimates arc subject lo statistical imprecision. For example, 
one could construct an upj^r and lower bound of a 90 percent 
confidence interval for the net impact of JTPA on postprogram 
earnings by adding and subtracting ! .% multiplied by the .standard 
error of the JTPA dummy variable to ihe estimated JTPA cocfticicnl. 
By adjusting both the upper and lower bounds for the fringe benefits, 
one would then obtain an estimate of the upper and lower bounds for 
the increase in postprogram output due to JTPA. Upper and lower 
bounds for the marginal cost of JTPA can also be obtained by 
creating a *X) percent confidence interval around the appropriate 
regression coefficient {i.e.. adding and subtracting 1.% multiplied 
by the .standard error of the estimated coefficient of Ihe number of 
adult men/women terminecs in the cost equation). Then, by choos- 
ing ditTereni combinations (e.g.. upper lK)und for benefits and lower 
bound for costs, lower btjund for benefits and upper bound for costs ). 
one can provide useful information on how sensitive the benefit-cost 
ratios are to alternative assumptions. 
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These sensitivity analyses, in combination with Ihe main bcnefit-cosl 
results, should pmvidc useful infomiation on the cffeclivcness of JTPA 
programs. 

Relationship Aim>ng Evaluation Mmlels 

Although much could be learned from implementing only the net 
impact model, considerable complementary information can be obtained 
by implementing the prrxrcss analysis and gross impact models described 
in the companion chapters. In this amduding section, we briefly 
indicate how iIk? net impact model relates to the other analytic ap- 
proaclKrs developed to assist states and SDAs in understanding the 
operatioas and impacts of their JTPA projects. 

The net impact and gross impact analy.scs arc quite complementary. 
Although both models are desigiK^ to address pix>gram etTectiveness 
questions, they diftcr in terms of the types of evaluation questions that 
can be answered, the range of outcome measures of interest, and the types 
of comparisons being made. For exiunple. the net impact analysis is 
Hn^ited to adults only, and because it relies exclusively on administnitive 
data stnirees. there aro relatively few outcome measures to examine, and 
only a limited number of personal characteristics can be included in the 
analy.sis. 

On the other hand the gross impact analysis can include youths as well 
as adults, an expanded set of labor market outcomes, and additional 
personal characteristics. As such, the gross impact model can be used to 
address certain relative effectiveness questions for youths served by the 
JTPA and can po.ssibly provide information on the mechanisms through 
which the JTPA affects adults* earnings and welfare dependency. 
Because of the additional outcomes available, a gross impact analysis 
may be able to provide some evidence on whether the earnings changes 
are due to changes in wage rates, changes in hours woiiced per week, or 
changes in weeks worked per year, although a comparison group is 
necessary to provide definitive evidence on these issues. 

In addition to providing complementaiy information on difierent 
subgroups and outcome measures, infomiation from the gross and i^t 
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impact models may shed light on important methodological issues 
affecting tte validity of analyses of social-prt^ram imj»ct.s in general. 
For example, the gross impact model can use imerview-rejwned earn- 
ings, whereas the net impact model rclics on Ul Wage Records to create 
measures of earnings. There are advantages and disadvantages to both 
approaches. By estimating gross impacts, using the net impact design 
and with the same samples of participants and comparison group mem- 
bers, <me can provide evidence on the extent to which the impact results 
are sensitive to the use of the different data sources. In addition, because 
the gross impact model has an expanded .set of independ^t variables 
available, by implementing both models using the same analysis samples 
it is possible to get some idea of whether the net impact estimates are 
sensitive to these omitted variables — an important statistical issue. Such 
compariscms provide important information on the limitations of the 
different analyses and indicate other independent variables or outcome 
measures that would be imjwrtant for subsequent program analysis. 

An SDA process analysis is a very important source of information for 
adding flesh and conceptual relevance to the net impact model. Because 
of the inherent limitations of the nonexperimental approach in estimating 
program net impacts, an SDA process analysis is a necessary first .step to 
a valid net imj^ct analysis. In j^icular. because the validity of the net 
impact results rests cm the similarity of the pa;ticipanl and comparison 
groups selected, it is critical to understand the JTPA participation- 
selection process, the factors that govern the assignment of participants 
to program activities, and differences in the content and recording of 
program activities across SDAs, A process analysis offers the following 
information for the net impact model: 

1 . It will provide a detailed description of the criteria (explicit and 
implicit) used by SDAs and their subcontractors in screening JTPA 
applicants to choose individuals for program participation. As such, 
the process analysis will yield imporiant insights into the type and 
extent of "creaming" that occurs and the likely differences that may 
exist between participants and comparison group members that are 
not possible to control for in the net impact model, 

2. It will include a detailed description of the procedures followed in 
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assigning participants to prograni activities. It will reflect whether 
more-advantaged participants are assigned to specific pn[>grani 
tivities, whether all participants arc first assigned to JSA. and 
whether only thase who aie not itnmediately placed are subsequently 
assigned to CT or OJT. This informaticm helps in determining 
wheUicr the estimated net impacts by program activity are likely to 
accurately leflecl the relative effectiveness of difTcrent activities, or 
n^rely represeni the fact that more-advantaged individuals arc 
assigned to certain activities, while less-advantaged individuals are 
assigned to other activities. 

3. It can identify major differences in the content of program activities 
across SDAs^ as well as differences in the ways in which similar 
program activities are recorded in the JTPA MIS. This infomiaticMi 
is useful in developing meaningful, consistent measurL's of program 
activities across SDAs. 

4. It will identify variables to include in the mtxicl. For example, it can 
identify SDA characteristics, such as servicx* delivei^ strategies, 
which are quantifiable md differ across SDAs. so that they can be 
included in the mixlel to test whether the net impact of JTPA 
significantly differs acn>ss these dimensions. 

In addition to benefiting from the SDA process analysis, note that the 
net impact model may also produce information that would be of interest 
to a process analysis. For example, the net impact model may indicate 
that after adju.sting for differences in jxirticipant characteristics and local 
labor market conditions, the net impact of JTPA is considerably different 
in some SDAs than in others. The process analysis can then examine in 
detail what it is about the specific SDAs that accounts for such differ- 
emres. If measures of specific SDA attributes that are potentially 
responsible for the different i^t impacts can be developed, they can be 
included in subsequent iKl impact models to determine whether they 
account for the different net impact estimates across SDAs. Such 
ongoing interaction between the process and net impact analyses high- 
lights tte complementary nature of the two analytic approaches and 
should result in an improved understanding of the facnors that affect 
program effectiveness. 
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.Summarv 

We have tlcscritHrd a general appuKich to examining ihe nci impact of 
a social program and illustrated this approach with an in-deplh descrip- 
tion of how lo estimate the net impact of employment and Lraining 
programs for ecoiwmically dismivantaged individuals funded under 
Title II- A of the Job Training Partnership Act. Although the details of 
the net impact evaluation motfcl have been tailored to a specific social 
program, the key elements of the approach have broad applicability to 
other siK-ial pn)grams. In jwirticular, any net impact analysis approach 
must include a ci>ncepfua! finmework, a research design, and plms for 
data collection. priKH^ssing. and analysis to answer questions posed in the 
conceptual framework. Morci>ver. because a major issue in any social 
pn>grani evaluation is the comparability of jnulicipant and comparison 
groups, it is likely thai the concern over selection bias will always be 
present and the statistical methixis described in this chapter are useful in 
dealing with this potcmial problem. 

Although ihe results of a valid nei impact analysis can provide very 
useful information cm the exlenl lo which participants overall are better 
off as a result of receiving program services (and, potentially, which 
panicipiint subgroups benefit most), additional infomiation is required 
to determine whether a social program is an efficient use of public 
resiiurees. Specifically, one must use the results ohrtained from the net 
impact analysis and other analyses to develop measures of total program 
benefits, and compare the total benefits with the costs of the program. 
Although the costs involvc*d in conducting such analyses as part of an 
ongoing program evaluation effort may be high in the initial stages, onc^ 
the system is in place the costs should decline considerably and the 
benefits from the evahiatt<^ should be substantial. 
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1. rs*r jxampic, i»w ol the tmp^n w»Mn» that the li^ Onps cUiH:utk>n ami matknuil traming 
pi^nim Un ^mi^gcd > hecn able ui a wul twigi!! cuU during rcicni yirar^, ck^ih* its 
ei^m?mdy high com per parliwifvmf > ibc m sulahiiity of Cim^dendilc an4:;uvh infcmnaium sndkrming 
thai iwps immthm recriv* iimg-tf rm mmcrniH* am) mwKrwmimitc hnwfits fnwn thir ptDgnun, 

2. Ottf cxccptitm is the c^per ina'nial evalu^km ul tTnplii>7mrm iawJ tminmg pn^ramis fiv aduith 
and yixjih^ limd«} urakrr tte hib I fftining Parinen^iip Act thai v^^s undtfitakcn hy ihi: IIS, Dcpan- 
HKmi of* Lalw tn IWte. This m:i in^psKrt study invi^vcs ^ptuim;acK 15 |:Ki^rmn i^^nHors 
nalicnwKkr. will 4.ml appit>ximdH:ly S^f nvliic^. vmd will he amtpkted in ! WI W2. 

3. Given ih»i the miirfcl i> u»>ahkr aroj fm>vKie?i valid rmxHs im tmpoitam posiprwgniRi iH3tcome^. 

acrt^K ^late% ^ is hkely to rwur. This will tna^imi^* the incf uiness of the inftHirtaticm <^ined lt>>m 
my hin^k imsdysis hy ^]itL*miing all smc*^' knowkrdgi: of wto is kmrnn ah(H3i the rlfetiivmsj^ 4»f 
ctnpU^>Tmmi m^H tminstig pn^gnons amcmg difftmrnt simt cnviitmmenis. 

4. The pnmitrv trxctr|«rons tiKluck l>icKjRson. jt^nMW. and We*i \ I V87a) h hti cxominrd C'ETA 
lie! impfiiis on ihc ixfmpcimrnt s uf comings* im.* Imling whether emplo>ied. the htnit ]y ^ age rate . hmrs 
uwked per **eek. and \*ceks linked per yvar. and Bassi ei al. i 1 MH4 ). ^ ho oImi e^minned the jmpja I 
vi CKTA pn>gnims im welfare i^pendeiKy 

5. infonmtum \m s^m^icm eammgv imp^Hris {iccumng within k*s,s thim a year can iwly he 
jWTnvided thnnigh priinan duta aillectsmt ef tints, or ihnmgh the use of I I Wage Rei^^nds. 

h. Beeause the net iinpmri nwdel is hssed exelusively Km available adminisirattve data, the 
cDiKepttial hBnte^i^ is m large imn daianietennined. Huv^vver. t\xn thiHigh tfv viHwepiual 
framewwk isionsirained b> the fcatuit's of availabk fiUiie daia Inues. virtually all ptievitius natiimal 
studies of the net mipm:ts iff en5pk>yi«mt and trainii^ fmi^tr^tvs summah/ed in the evaiuaf tim guule 
shaw several of the limitmions of the net hnfmi iruidel ikscnhed here. 

7. AlthiHtgh these adrntnistrative data sounes aa* vcr> tnexpenstve. partkuiarh ivlaf ive to the 
co&tj^ of survey data. iHMitnviii) data jnivessing vmsts imiM he inetmed u> aecevs if^ af^^^iate 
leeords fnim the systctn. Depemiing on the si/e of files in the st;rte. these ci>si«i cf*aW range from 
several huudnrd ck^Iiars lo several thmisaiuJ dollars tor Trmv\ per run. Mimrover, prictf to 4>htaining 
these data, tt will he nceessar> to meet an> state nrquinfments amireming data eonfickntialfty. and 
10 take steps u> maintain data confldentialiiy <e.g.. rrnnive all ideniifying infiMmaiitm and cteate 
unique tdrattfarr Uif an*il>^is purposes 

H. Sonw states that an: m>t formally wage-reporting states tuive 4H^mpanihle earnings records 
avaiiahle that arc maiiuained hy their Departn^nis of Kevenmr and nntld he used in the analysis if 
the necessary mteragctKy agneenumts are mmk. 

^. Fwexampk. in some stiites. V] Wage Records do not imiliu^eamsngsforthefolkming types 
of emptoyecs; certain ciNTwate officers, church empJoym. incii vidiials paid exclusively im ctmrniis- 
sion dtOTtestrcs wIki earn less than a certain stnount per qw^^ tc.g.. $ I ,VL\0 h railft>ad eniplo>ve^. 
cmpliiyecs of smaU agricultural firms, casual lahiwrs. and certain harhers or ccmnetokff is])». As a 
result i>f ihc*se coverage gaps, approximately ^\ peaent of all stale %ages are general!) mclu^kd in 
the VI Wage Reconis 'iaia hase. 

JO. }Hn example, stales that ar, very interested in devek^ng net imp^t estimates Un J TV A Title 
ll-A youth pn^grams might amsiikr miplciTwnting an experin^ntal design. AhermUively. states 
might consiikr ci>nducting irel^ively expensive) iniervH'ws {mrttcipanis and ctwnpWTMWi gnnip 



126 Evaluating Social Pn>blem«i 



n^tnherK u% cnlted the deimkd prtftn^^mi and fMi«»^^nm) mplmment and %*ho4>lmg diUa 
nec£H&af> k»'ir liable ansuy sis. In ett}^ctt!ie. the geiHrroln^Kemh design mni analyst 
fatter ni thin chapier mikJ he ft>Iiowed. Smie^ iMwaed m Mwrh apjwwachev tKiw^Tr. ^hmhi fina 
ctmKud enq^dymeitt aroi naininj^ nrMOTrt^ who arc knowiedgea^e in experimental ^ign ivsi^ 
ami qatsxkmtmire drvelojmKni i<> aviwd the piifaiK fhm have [Hagu^ pievkni* Miuiks. 

1 1 - If , hcwwfvcr. wwi esipenetiLT mmik iJifn^r fm»grw« mriivit> hi lused extensively in a pankrukr 
Ktatc M) fhm the simiple sires anr suffKa^m lo siippon f^ech^e net tmj^wt e?»timaiev, ii would lAm he 
poioiiMe fblkiu ihe i^her pn^vduiv* tiiHlii^ laier in ihis chopier m e^imate ifw ne! impaciH fiv 
Ihifi activiiy. 

12. On the (Hher haml, an enn^lJre-KiMrd sampfe fran^ Mwiie diJiiadv^^cv T«» avokl 
excluding k)ng>ienn paitkipantH Imm ti>e ;m«lyMN, an enn^llee sample frame coii^e^ a (fclay m 
.mdyMft ftndmgSk t^loii ve to a brrminee-hatied ^^mple. In ac^iiicm* hecAutie a given grmip of enmlke^ 
may lermmate mro^s diffenem i|umter\« with a design it vh mure difTti uIt to eMimate earnings 
impaet^ thai cme^^poiKi ^ s^iecifit tinK pernKlsv after prt^am lermimjtkm. Mich ah the three-nHinth 
period folUtwbig the quatter after termitiatkHi. I lowever. aliemali ve i^ample ftamef^ differ frmn cHher 
pfoMeniH that aie nnn^ Mrvere. vrMkh kd m to xhc il^shm to u«»e an eitnillec-ha»^ i^ampk. 

1 \ The limited amtnim of mbning data is. in part, a renecittm of the pwedua*** u><rd by many 
agemk^ to av^tgn "iferfmili" valuta when data an? mivstng. Such prucedurcK kad to meaMiren^ 
vmn, which can also imrodittx analytivai ctmtplicalions as discusscti later* 

1 4. It IS ck'simMe to examine the mivsing data )m^lems hefore imking a final det'iston on whether 
to I- xclude siarh cases f n wi the im»lysi% *amphr . If the data item% mt missing for ntmkmi reasons, t hen 
no harm is tkme hy tmiiftmg such ca^rs fttm tlu' panicijmnt gnnip. If, howrvcr. it is determiwi that 
the reasi^ the data are missing is systematically lated to the impmi of the imigram {e.g.. individwds 
with missing data m length nt stay dnif^ied intt of the fm^gram ami Htrmr k*&s likely to hp plmrd u this 
^t>yld reduce the internal validity i4 the overall ;inalysiN. Thus, some simple compisisons of the 
characteristics of pmiicipunts with missing ito and p^icspsinis i^uh cumpkrte daui im fHt)gram 
srrviiTs w$|l he perfoimed as deHTibed in suhsct}i»em seiikms hefmr b final tkcisit^ is made to 
exclude cases with missing data inm\ the analysis sample. 

15. example, in scwne SDAs ut«nen a>m|wised as little as 2S pea'vni «if adult JTPA icrmmees 
in PY H4. while in other SDAs women wene over W peawt of all mlult lerminees in Title H-A 
^^pams during this period. 

16. According to the JTPA legjslatnw. to he eligible for 1 itle H-A pn^gnum. adults must he 22 
years of age or older and he eitHWrnicaHy disads antaged. Tt^ ad shinild he ci^sulied for tlw exact 
definition of "ectmwuiiajjy disadvantageti** To the extent possible, the comfmriscm group shrntld 
only inclutk individuals who mc€\ the explictt ehgjtHltty cnterui and who are similar U) padkipants 
on chaim'terisiics emphasized in the legssk^iiin. 

17. Based on a croipariscm erf ES and JITA data collected m selected states, the following 
individual charaHcristics are generally cimtpar^y measurcsi: age. rakc/cthnicity fuhltc. black. 
Htspsmc. Amerkrsm Indian^Alaskan Native. Assan/i^icifk: Islajickn. edmraitcm (whether received 
high school degree or equivalent), handicapped status Cwtufthcr has physical nnental impasrmeru 
thai ih a sub9«lantial handk:^ to emptoymeniK occupi^ton <prtmary DOT code of f»eviut& jiib), 
veleian fOatus t a vctermi. a V »etnam-eni veteran, m-emfy separated » a di sabhfd v^ 

recipkfnt. WIN legtstram, and ecom^ically disadsaniaged status. In ackiitkm. j«tfpmgram measunrs 
of L'l Wage Rectwds, AFDC' grants, and whether »V\ recip^nt will also he availahk and t^mparahly 
measimrd for both panicipantsaiMi a>m|wis£m grmip memberv Alti^gh this list is nut as cc^pleie 
as one would ideally like - nrKasuit^ of marital suitus. family si/e. dependent chikhen. eX'«?fTen<k?r 
status, limned English speakuig ability, and cktai^d d^ tyn pepn>gram empkiyment experiences 
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ane nrt avail^^-^ft im^ be fecogni?3cd that most of tteic chara^eriMiis si^trre unavailri^k Id pre- 
vioitt nalional midics of ihe imjwci CTifHoymem and traimng pft^^m. As sitth, Ihts is ik« a 
Ibnitation dmr is specific w ihe modei dei^bed tot?. 

mmurenicni error into ih}% vmstk. N<« rnily did ES jaafT pirvicHJiay have no ma! inraitivc 
to accuraaHy rcrad il» fian» erf ite apjrfi«mt <i.c., gince ES scrvk^ ck) noi dqsemi wt^ther a 
pemjn e ecomsmically cfinadvaimgcd), but Ihcy now have even less iiKTiUivc lo «k) $o. As* a resuh, 
h likely dim ES cmx% ixrukr rtpM hmhig nuch ^Ikani*. Thu», to iIjc extent thM oti!> ES 
n^wtrams w to are rwroided ^ msiomically dis^imt^j^ are included in i!k sample frmne fi^ the 
comparison gtmip*. xJt^u statm* ^hmild be mcawaed n:»ion;^y aauratdy. which wiJI minimis 
ctHnpUcmions due to meaKureni^ enw. 

19. Several midks exclusk frtmi the cwi^wism group individuals with verj high preprogram 
earnings wl» were clearly ineligible to paiticipaie in employment ami tniining programs (e.g.. 
Dickinjion, Jdinison, and We« { !«86); Westat ( \9M)). By m^hing participai^ and compamcw 
group members on eo^Kmikrally di^^antaf^ sratujs however. Mich mUitkmal excluf^ms nluHiid 
no be nccemry. Nc^ ^ if the eccHwmically dismivaniaged st^us variable is ncM availabte 
in some sia»s fw ES registramR, ll»n piocedutex io exclude c^ with high jw^rogr^n earnings 
muM be implen^nled a.^ describe later m secticm. 

20. The soatistical power of any hypiMl^is ^ relates to ttK? likelibo^ 

<tf inoHTec! coikIusicwl TNr power of a tei^ cMcems wh^ ts called a Type 11 eiror. or tnconectly 
accqxing the mil! hypc^hcsis {e.g.. tto there are no signifKMi differences in earnings between 
pnagran partkripams and ihe cvmimriaon gvcmp) wtei tlw null hypoth^is is false. Altertiati wly, the 
statistical power is the pfobshilily of ckneclirHs an effect lat ^ chosen signifurance level) when the 
effect of the specifml siw, in fact, exists Oe., it is I minus the fm>babiJiiy of making a Type II enor). 
Because 0» ptoMnlity of making a Type 11 error dec]ms% as sMnpk si^ flrxn^^ies. larger samj4es 
are used to minimise Type II errors and maximixe the powr of the test. 

21 . With sikA a s^pte design we climate th^ it will be possibk? to delect a^roxinnitely a five 
< six ) pereentage point inipact cm emiings for »jult nK*n ( aduh 

significance level, Tto is, ch» wmjk! fwve 90 jmnxm power al a 0. 10 Mgmfarance level of ik^nmg 
an oveiall net increase in pankipai^' eambgs of as sn^l as five or six pereei^age points, 

22. *^i»ingemoJbnemslnmithcpiorpogramye^ 

it may be necessary to adjust estimated enroHn^s to rcfkd changes in real piogram lescnirees. that 
is, changes in fetkroi aliocatiom adjusted ftir inflation. Such adjusimems can be made using 
InfwnraikMi on the percefuage ch^e in program expniditures typically inctared for a given 
poccniage charge m the number of JTPA pmikipifflts, whkrh can be c^^ained fftnn the cost anal>iJis 
ikseribed later, 

23. Note thai because eligibility fw ES servfces does not depend <m ea^K^nkially disadvantaged 
ftlatuii, d is likely that this indicant is measured with much more cnxw for cmiporison group mcmteni 
than fw JTPA partkipanu, Ffowever. we expect ilmi the otot is mine likely to be ui jctemifymg 
some registrants as disadvantaged who in fad arc. Thus, by only retaining in the cttmjwmon gnjup 
those ES registrants wte are indicated to be ecffliomically disadvmaaged, tlw gimips sknild be 
reaswiably ccsnparabk «i this ctoncnsiim. 

24. Abhoi^ this would help to en^ue similarity in temis of maxi nwm earnings in ite prcpmgmm 
period, in the absence of data on economically disadvantaged st^us it is also desffsbte to ra^ the 
samples more closely in twms of the time pattern ami levels of fweprogram earnings. For example, 
teed m the jHeprogrmn pattern of j^kripams' eamit^ one could err «e spccifM: cells that are 
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mvnmJ!y cxlIumvc ami exhaustive of al! possibilitk^s. amJ ihL-n sclixx arnipiiriMVi gnnip membcix 
fhwi \ht^ cells m mah:h ihc disiritHition nf partrci;^!N. 

25. Vsm$ quancrly valuc\ 4»f tlNr 81.S Consume Prit c imhn Un all I Than Wupt* and &iLar>' 
Hamm. uiw wiwld deflaUr liiiv i^te ) the valu^ of the variables expjvs^ed in mmiimil doU^ lermiv hy 
Ihc value of the ustce imkx in the san^ calemlar quarter, ami cieate nK-asures c»f rea! earnings and teat 
AFDC grams reecived in each jwj«x>gram i|uarfer mi real l^I benefits aveived in the immediate 
prepti>gram quoiter. 

26. fnn-data items ;^imrd fimn ES JTPA applik-aiicw forms, ite range of a»xe|Hable vali^ ean 
be specif^ exatily , Thai ts. if hamiiva{7ped status is ctxied as 1 for ami 2 for mx then an> valwrs 
inhcr XhBSt I w 2 are clearly emw> that likely oecuned in entering il^ data into the MIS. I'nlesH such 
em^fs can K: a-adily ctiTTCCted using c^her inft^nnaiij^t im \iw file, they slwuld be set to a annnwo 
missing daa code te.g.. -M) and dealt with as fwi of the fnoceduies Un handling missing data. 

27. Hx example, tme ean use mean values, a htrt-deek or eokl-dcek ywxKedure, a aggression 
equation, or other iiUKe con^hrx methods to deal ith mivsing data pa^blems. In ger^rah as kmg as 
the leaser a variable is missmg is not correlated with the variaWes a'piesort ing im>gram partk:ipjaKW 
f e.g.,classaHwi trainrng. length of program participation K no bias is introiltieal in t^ estimate i^rxi 
fm^gram impacts, althimgh the standard emn- of the variabie that l^s K-en imputed is reduced aiul the 
pnecision of the estimated imp^urts is overstated. 

28. It %^tHild also be possible to estin^tc an OLH linear pn^bility mcnlel 4>f the likelihood of 
panicipaiing in JTPA todeiennine the major ditterentcs hetv^een thr two gamps. TTiat is. one ^^^Hi^^ 
estimate a fegies.sion eqttaiion with the ckpcmknt viable equal to 1 for JTPA partkipanis and 0 Un 
ciwnparistHi gamp members, ixnd the indepentkm vanaWes would he all measured characteristics 
included in the net impact mo(k*1 rk»scribed later in this sectiiHi. Tliis appam h has the advantage of 
estimating the independent inHueiKc of e;H:h measiued chan&neri.sttc. whik conmilling ior the 
influence of all tHher chaiacteristtcs. which eliminates the confininding effects of cHher vari^^les that 
may be pteseni when comparing mean characteristics. That is. a compuriM>n of mean characteristicji 
could indicate. example, that JTPA participants are nK>rr I i kely to be minoritkrs and lc%s editcated. 
whereas the legressi^i approach winild j^rcount few the difTeremres in educitfion h> race ami could 
revral that, after mliusting for dif feneiKes in race, thei^ ate no differences between particifmts and 
cumpariscm gn^ip members m terms of edmration levels. 

2**. As indicated in the second ami finiflh guidelitws for selecting indepemkni variables discussed 
above, the only changes mrce^sary in ti^ imkpendent varud^les in changing the depemiem viiriahle 
from the immediate prepro^am quarter to the imtnediate pn?pa>gram >'ear wmsid he to ensure that 
quarterly earnings ami AFIX' grants in tlw second, thini, aiKl fourth preprogram quarters w«e 
excluded aixi tl^ the unemploynKni rate was (fcHped for the entire preprogram year rather than just 
for the immediale i^mgrmn quarter. 

.^1. Most standard regirssic«i {^ograms allow to perform an F-rrst of such an hypothesis, and 
also proviik: the cakulafod F-statistu for the test. Under the assumpitw th;tf the mnr terms are 
noimalty di*»tntHJlcd. the test statiMsc follows Snttfccor's F-distritHiiit^i with r degrees of fteedtw in 
the nunKTatorand N-K degnres of freedwn in the dctKmiinator. where r is the number of rpsirkticsis 
bcii^ tested Ci4?., the number of irKicpcnctent variabtes that ha\'e been muhiplied tl»? JTPA 
dimsmy ). and N*K is the mrmber of degrees of freeckwi when no restnctkms are imposed C i.e., trtal 
san^le sij£ lessd^ number of variables in tl^equiftion). If il^ te.st statistic exceeded the critkral value 
for the specified level of significance, tf^ the null hypothesis would be rejected and we would 
conclude ±at the f^^program earnings iim* AFDC grants) equatiims for the two gauips are not similar. 

^ f . Note that individuals who sat still in tt^ pa^am in a given quarter should not be imrlrnkd when 
analyzing ti^ impact of JTPA on earnings or AFDC gnmls during tlmt quarter because Uieir eammgs 
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may be unuMiaily km ic.^., fur cia^«jiMvm iraining ji^ search av^Mdncc {^icipan^) w unusually 
h?gh (e.g.. for OJT partkipanisn which would Nas the esttrmied overall net inipacis, 

32. Prt:viou% CETA Miidies indk*Hic that pmgram m*t impact esSim»tes could be scmKrwhat 
senMtive to ihc e^cluiikm uf lung-lcrm {lartkripants. {nitnariiy becuuiie (be exclmicd imUvidiialK 
tended lo be in Public Sen ice nmplo>ineni pn>granis and wea' alw ays w utking { Dickinson. Johnnm. 
and West IWlb). Although this is gewralK iic^ likely to he the case in JTPA, this suggests tJuit. at 
a minhnum. stales shiTUid try ioi>hiain Icmger rofk>w -update to use in iKkJitfcmai analysii^ thm incit^*s 
alf kmg-ierm participants to examine how sensitive the lesulls arc to this issue. 

33. Because of the diffea*n! sample si/es involved in anal) zing impwcis ditfea'nt poMi^igiam 
periods the prccisk>n of the essiinmed 1 2-nHmth $wi im;^Hls w ill be krs?^ than ihe precisk^n itf the 
estimated impacts iwer si thn^'-nionth periixt. 

34. It niay also be of imciest to determim.' how the effectivei^s of the JTPA differs aoKHig SD As, 
This can he determined through in.I;:i*jng separate SDA-participant interm-tion terms using the 
funeral af3^mch tkscrihcd below . 

35. h should be mned diat. in print4ple. simi !;«' analyses ctnild be perhtrmed to determine whr iber 
net imp^ts vary acn^ss kical labor n^irket umdiiions. However, because itie laixir market variables 
w oukl take on the same vali^ for all penii^ in the same local aR'a in a gi^^n time perkxi theie is 
likely to bt* suffii ienl variatitm to c^ain precise estimates i^htw pmgram impacts vary acnwis local 
labor market ciMiditions, except in large stales, with many SDAs. am! where theie are considerable 
differences in labtn- niarket ct)nditions across SDAs. 

36. To fimnally lest whether tin* program net impacts differ signific^tly acn^ss the gn^ps of 
imeirst. an f-'- is used. In this case, the test sunslic folbw s an F<r.N-K ) di.stntmtion. where r is the 
number i>f n^slrktiims impi^ by the bask model iequal to ihc number of gnnips minus tmc), and 
N'K is the number of degrees of ^ieedi>m in the basic impact model. The hypott^Ls that the 
impacts do not vary at mss the groups of imenrsi (e.g., iy:ross r^ial gniups ) wouki be rejected fw r 
= 2 and sufficjemly hularsampk- si at theO.I>5 C(M)I ) significance kvel if the tesi siatistic e^ceetkd 
2.W(4.trfM. Mmtsumdardsol^ware analysis packages cakrulatc thisF .vrimA/n aspanof theaemlysis 
nm. 

37. If the chanicteris!k*% defining the sul^ruups of InteteM are not nufasured equally well for the 
partk ipant!) and compariMm group nicmbers. howe* cr. the subgrtHip impiK.ts will it^pj^^ately ne- 
fkrt ihese differences. Becaui^e tlic j^*sence of mtasuteinent error in an iixkpendem variable biases 
its eslnnated cwfTicient downward, if the annnjnt of measurenx'nt cmur m a sut^ioup '-haractcns^k 
were gnrater in liie JTPA sample, for e;^ainple. than in tin? ctMnparison group, the efTect of that 
charw.'te'c^iic on tfu.* cnttctm^ variable wiHild be smaller m the JTPA sample than in ttie ES registrant 
s^pk. .he in^rsctKMi term woukf mapprojMialely pkrk up luich a difference and misle^ngfy 
indicate that JTPA impacts wert! smaller fi^ indivkiuals with that charfH:teri.sttc. 

ReductiunK in transfer payments (e.g.. AFW grants) do rm lewesent a benefit from the social 
perspective, since the increased benefn it) taxpayers is otTset by the k)ss of income to recipienis. and 
tlwrc is theiefcm nochange in tl^ rcivnuves available to sockrty m a whok. Thus, including estimated 
benefits from reduciiims 'u\ welfare depemkncy due to JTTA would involve a ikntbte counting of 
benefits. 

39. AlthtHJgh very unlikely, in the extame. the prt^gram could produce no net sncrcuse in output 
de^ite large incieases in parikipants' post|Mt>gram eammgs by simply nrshulfling^^ fivm ikm?- 
participants to participants. 
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Evaluating Gross 
Program Outcomes 

Car! SimpM>n 
S(Hiohi(y DefHirtme$U 
Western WashinfiUm Universiiy 

S(mie sdentLsfs read fo (he dijfii uifies esfaMisfmfi cause and 
effect by withdrawing into (heir shells and refusing to say that the 
relationships (hey find are anything more than correlatif^ns , . . But 
decisionmakers cannot avoid nuiking jndgmein . . . The dn isitm 
maker want to know wlutt lo ciiange so thai he can achieve (he effect 
he wants. 

Julian Simon and Paul Burstein 

Basiv Research Merhtnts in SiHial St ivm e 



General Concepte and Methods in 
Gross Impact Evaluation 

Managers of nearly all organizations — human services and private 
sector businesses alike — examir^ the outcomes of iheir efforts and 
compare their own SK:h!evements with those of other similar organiza- 
tions. Descriptions of outcomes and ccmiparisons across oi^anizations 
can be valuable management tools, but can also be misleading. The gross 
imi^ evaluatiOT model offers guidance for maximizing the usefulness 
of these management tools while avoiding errors commonly encountered 
in the attempt. As Blalock argues in chapter K we want to facilitate 
evaluations that remain closely tied to management infomialion mxds. 
but go beyond program mcmitoring^ both in terms of gc^Js aiKi methods. 

The Perspective Taken in Conducting Gross Impact Evaluatfons 

Evaluation research is often viewed as remote from service (fcli very — 
as serving distant purposes or as serving no purpose. However, the 
analysis of data on services and outcomes can be a valuable management 
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tool guiding program development efforts. In dddiiion, with the advent 
of computeri^ management systems, such analysis has become fea- 
sible in the majority of human «jrvices organizations as well as in the for- 
profit sector. Data systems put in i/iace to facilitate record-keeping and 
report generation can be extended efficiently to provick a basis for the 
analysis of service quality and effectiveiKss. Perhaps these factors help 
explain why a recent survey finds many service delivery areas (SDAs) in 
the Job Traiulng Partnership Act (JTPA) system expressing a special 
interest in systematic self-analysis (Seattle-King County 1985). 

Systenuitic desvriptians of program outcomes can focus policy plan- 
ning discussions. Further, amlysis that cimpares the effetts of program 
alternatives can identify strong and weak areas of current services, in 
terms of their impacts on outcomes. The ability to focus change efforts 
on low performance ar^as and to identify high pcrformaiKe approaches 
as models for planning can enable a continuous improvement of services. 
It amounts to "technological advance" for service organizations, where 
effective technology — knowing what transformations produce desired 
outputs — has been dilTicult to develop. This capacity to direct change 
intelligently not only improves program services, but also provides staff 
with a sense of efficacy— the sense that they are able to affect the quality 
of their own woric. Staff burnout has been identified as an ongoing 
problem in job training organizations (Franklin and Ripley 1984). One 
partial solution is putting the tools for more effective management in the 
hands of local staff. 

Even where a foundation of previous research has been laid, analyses 
by local or state service delivery systems is valuable. Management 
decisions must be specific: shall we implement services this way or that, 
place participants with this type of traimr or that, deliver this set of 
services or that? TIkj local context determii^s which alternatives arc 
available and also the xlative effectiveness of e^h. Thus, to apply 
national research findings to local settings, while the only recourse in the 
absence of better information, is a less reliable management guide than 
developing local knowledge. For example, Wilms (1980) found no 
difference between public and proprietary vocational trainei^ in South- 
em California, while Simpson ( 1982) found public schools substantially 



ERIC 



} 4 3 



Evaluating Gross Pnigram Outcomes 1 35 

more effective than proprietaiy schools forCETA {^icipants in Wash- 
ington State* To make policy decisions in eitter area based on reseairh 
in tte caher area would be unfortunate. 

Grass impact analysis is the study of program outcomes among 
program participants only. It resembles i^t imi:^ct analysis in that it 
focuses on outcomes and their probable causes^ but it differs from i^t 
imjimct analysis in that no comparison group of nonparticipants is 
involved. Gross imj^t analysis provides quantitatively reliable knowl- 
edge about program quality md effectiveness, with the goal of guiding 
program development. The method builds efficiently on already-exist- 
ing dam collections adding data elements as required, to improve validity 
at a reasonable cost. Posiprogram follow-up surveys are used to measure 
program outcomes. I^pgram qualities tested for their possible influence 
on outcon^ levels arc measured using available management informa- 
tics systems (MIS) and^ where applicable^ by collecting additional data 
describing services delivered to specific individuals and forms of pro- 
gram implementation developed by various service providers. 

Questions Gross Impact Analysis Cannot Address 

Among the wicte range of possible imp^ts of any human services 
program, only a smaller set can be addressed using the gross im^^ct 
ai^rc^h. Several important society-wide goals of employir^t-related 
im)grams arc essentially impossible to study definitively, because legis- 
lation that improves the situation of some individuals may be creating or 
overiooking problems among other individuals. Tl^se impacts iiK:lude 
(1) increasing national productivity, (2) reducing total nati<mal unem- 
ployment, (3) reducing aver^ job turnover time, and (4) improving the 
skill level and, therefore, the flexibility of the overall labor force. 

In ^iticm, gross impact analysis cannot draw any conclusions 
ccmceming the types or degree of change caused by participation in a 
particular program. Hiis question can only be Pressed by net imfmct 
studies, which compare program participants with similar individuals 
who did not participate in the pn^ram (see Jcdmson and Stromsdorfer, 
chapter 2)* Gross outcomes refer to total postprc^ram outcon^s; net 
imp^ts estimate the proportion of total outcomes that may be uniquely 
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attributiKl lo {micipaiion in the program interventtcHi. This means that 
gross impact studies cannot estimate the extern to which panicipation in 
a pn^nun changes individuals, the cost-effectiveness of a {^gram« the 
time it takes for participants to i^pay the cost of a program in taxes 
geiKratcd by program success^ or the inipact of tlw program on reducing 
other costs such as welfare supports. 

Questions Gross Impact Analysis Can Address 

There are two bn^d categories of analysis gi>als for which the gross 
impact approach is well-suited: ( 1 ) Jestrihing a broad range of program 
mwomes, and (2) estimating the unique imfkici on outcomes produced 
by alternative metiunis of delivering services. The statistical assump- 
tions underlying these and the power of the occlusions which can be 
drawn from them are so different that the remainder of this chapter will 
refer to them sefwately, as gross outcomes analysis and differential 
impact analysis. The finst term avoids the word impact as a reminder that 
no cause-and-effecl impact can be e^:!imaled using descriptive analysis. 

The Description of Gross Outcomes 

The description of gross program outcomes does not allow the eval- 
uator to infer causation — to assume that the program or some aspect of 
the program is responsible for tl^ outcomes observed- Tliis type of 
analysis is well-suited to describing a wiite range of outcomes for 
jmrticipants, employers, or others, with results available in a relatively 
rfiort time. Descriptive findings in themselves imply no success or 
failure, but can be evaluated against managen»' expectations. Descriptive 
data may also help establish reasonable new baseline expectations for 
outcomes not previously measured systematically. Discriptive gross 
outcomes can also be used as tools to identify prc^leni areas that (kserve 
more detailed analysis- Finally, tl^ ability to measure relatively numer- 
ous ami (ktail^ outcomes provides a way to describe the range of 
program outc(Mnes, and how programs are achieving their impacts. For 
example, job training program outcomes can include such issues as 
wheUier employment is training-related, whether fringe benefits are 
IHiovi<ted, whether promotions are likely, and whether empIoyen> :irc 
satisfied with the progimns. 
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Gross; outcomes may be described for one office, one organization 
with multiple offices, or a system iiKluding more than one service 
delivery organization. However, where more than one organization ot 
office is (kscribed, caution must be exercised in interpreting any com- 
parisons that are made. Outcome comparisons tend io be interpreted by 
research consumers as if caused by differences in program effective- 
ness — an error, given the limitaticMis of descriptive analysis. 

The careful de?«:ription of program outa>mes is often a necessary 
element of process evaluations (discussed by Grembowski in chapter 4 
of this volume). Two fcx;i of gross impaQl evaluations— broadening the 
scope of outcomes measured and exercising technical care during 
measurement — are therefoi^ particularly valuable to process evalu- 
ations. By the same token, a prcK^ess analysis can be extremely valuable 
in identifying the particular outcomes that are appropriate to measure as 
part of a gross impact evaluation for a given organization. 

Differential Impact Analysis 

The second type of analysis proposed here docs involve the estimation 
of cause and effect. Differential impact analysis is a method for rigorously 
comparing program variants — alternative service strategies and alter- 
native approaches to implementing service delivery. Different program 
services and implementation forms can be compared to assess whether, 
other things equal, one or more alternatives are more effective tfwn 
others in producing desired outcomes. That is, the unique impact of each 
program variant can be estimated in comparison to all other program 
variants being used in a program during the analysis. Participants 
experiencing each program variant act as a comparison group for those 
experiencing other variants. This opens the way to a wide range of 
analysis questions that might be asked by managers. The specific 
questions depend on what program variations exist in a particular service 
delivery system, and cm the areas in which managers arc most interested 
in developing information. Tlie term program variants is used here as 
shorthand to include all existing program alternatives in services a.s- 
signed to individual clients and in forms of implementation found among 
service providers in the service delivery system. 
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Within Ui^ coastraints of namplc ^\m^ the^ same quest iom can be 
a^oi ioT particular populations partivijhints: which trcalment moctes 
are most effective for target group A. and which for group B?* Compari- 
sons can also be made cunong service pnniders. This means thai states 
can improve the reliability and meaningfulness of com|wisons made 
among local ageiK^es, and can also identify cs{^ially valuable direc- 
tions for prognun technical assistance effons, as Icmg as the influeiKres 
of enviixmmental conditions and other important difTerenccs among 
agencies are taken into account. 

Differential impact analysis is especially well-suited to prognun 
development efforts because of its abilitj to identify program variants 
thai influence the program's ability to produce desired outcomes within 
the limits of the [Hirticular clients it senses and within the range of 
treatments available to it. It caniK)t tell us whether a prpgi^ is worth 
retaining. However, given that a program exists, it can point the way 
toward making it operate more efTectiveiy. 

Differential impact analysis recognizes that the most serious threat to 
reliable comparisons among program variants is selection bias. Clients 
wlw select or are selected for different program variants often differ from 
each other in ways that influence probable postprogram success. The 
impact of such client tockgixMiiKi differences must be mrounted for 
befrae Uk? unique impact of program variiuits can be identified. Other- 
wise, estimates of program variants are biased. Although iw set of 
measures could ever estimate all selection effects, it is possible for 
analysts who know a particular service ctelivery system well to construct 
measures that will prevent a considerable proportion of the bias that can 
be caused by unmeasui^ selectitm. Each additional investment in such 
preventative steps improves the validity of the research findings. 

Tli^ second major threat to differential imp^t analysis is from 
confounded program variants — service strategies or program implem- 
entation modalities that completely overiap (me anoiter. If all clients 
assisted through a particular service provider are assigned services that 
differ in, for example, three ways from services given all other clients in 
a system, it is impossible to determine which of these three might account 
for a higher or lower success rate by that provider. TTie treatments overiap 
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so compictcly thai they cannot be separated simistically; ihcy arc con- 
founded with one anolhen 

Similarly, any time one service delivery organization implements 
program variants that arc entirely unique, the analyst encounters at least 
two factors that are confounded: whether participants experioiced the 
specific program variant in question, and whether they enrolled through 
the specific organization in question. Thus, completely overlapping 
implementation by more than one provide j> well as completely unique 
implementation by any prt>vider constitute confounded prognmi vari- 
ants, which make fully accurate differential impact analysis impossible. 
This means that the differential impact analysis of treatment variants 
becomes possible only when the treatment system being studied includes 
a sufficient number of service provickrs or .service tracks that implement 
services differently, but not completely differently, from each otter. 

The Types of Factors Measured During Gross Impact Evaluations 

Both descriptive outcome analysis and differential impact analysis 
require measures of program outcomes. These may be recorded by 
service providers, reported by participants durng follow-up .surveys, or 
reported by employers, ^l addition, differential impact analysis retiuires 
the measurement of program variants and "control variables." Program 
variants may be nKasured at the individual level, indicating which 
specific services each participant received, or at the 5^rvice provider 
level, indicating how programs are implemented for the average client. 
Control variables include a wide range of selection factors that can bias 
findings if excluded from the analysis. 

An overview of the various types of measures that may be involved in 
gross impact designs is piovicfed in exhibit X I . Measures are grouped 
according to the source of each measure, and the purpose each measure 
serves in an analysis (outcome, program variant, control variable). The 
intersecti<Mi of each type of outcome variable with each type of program 
variant .^hown in the exhibit indicates a major relationship studio using 
this approach. In exhibit 3 J, tlKse relationships are indicated by letters 
A through A^. Any given differential impact analysis involves the relation- 
ship between one outcome variable and some number of test variables 
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(program variants) along with the comml variables included to protect 
against selection bias or other souires of error. Within this siruciim:, each 
test variable represents an "hypothesized" i^ffcct on the outcome in 
question* 

Exhibit 3.1 also indicates the most likely sources for each type of 
measure, recognizing that measurement decisions depend on the luture 
of each service delivery system. Outcomes are measured through obser- 
vation of participants or relevant oilxrrs. Some outcomes arc observed by 
service providers, while others require sf^ial data collection. Surveys 
of service providers are ine\{X!nsive tecause their numbers are small 
comp^d to particif^ts; however, they can measure only implementa- 
tion variants* not individual service vari^its. Participant service lecords 
kept by service providers are both more reliable and less expensive than 
measures included during follow-up interviews of former participants. 

Itoturs ami Stmreet of M8»im Hiit V^qr Be li^MiH 



Tte Pur)»n» of ^ Ifotttire titf RBMhm to JTPA 



tDOQRMiVAmAHTS CoDtrots 

OUTCOMES FOR Unrtn sgaimt 

DotiSoyfco ftrtfeiimnti t^rs Impltmen^tioi! treotmoM Mas 

Survey oi sefvtts providers f J 

Participant treatment kept 

l)y serviw provldm D 6* H K 

Standard MIS til» A I 

Particij^ lo!low-up surveys B r • 

Foltow-up surwys of otliers, 

suet) as employers C E M 

Published data by i(K:alfty H 



* indnmiua) service treainftent records may aggreQated to mdicale typKrai agency patterns 

• • Sefected treatment variants (an be measured through participant foltow-up surveys, although 
the^ require retr(^pactt^^ re{K}fts 
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Variables measured as controis anainst bias can come from many 
sources, depending on the specific design of the analysis. Service 
providers* p^icipants« and others, such as employers or referral agen- 
cies, can provide valuable data concerning the selection pnKcss. Service 
providers also implement policies that affect selection bias. This does not 
mean the analysl should ask servio! providers for their interpretation of 
their own selection processes, Raihen it means that once the analyst has 
identified specific selection {Kiiicies and praciices likely to affect oui- 
comes« these will be measured most validly through reports fn^m those 
directly impleftumting or experiencing the pnK*ess. In addition, MIS fsles 
and published demographic and labor market data report standard 
variables knov^n to affect labor market success and are, ihereforc, 
necessary to include in diffe>eniial impact analyses. One sta*ngth of the 
gross impact approach is its flcxihU' ability mmsurv multiple imln a- 
tors of selection into purtividar sen ice treatments. 

The I^ign of (iroi^ Impact Kvaiuation Studies 

Nearly ail modem human services organizations have committed 
records of client characteristics at intake and services received to the 
electronic memory banks of micro- or minicomputers. The relatively 
ready access an analysi has to these MIS accords lays the initial base for 
inexpensive, yet valid, gross impact evaluation studies. Client popula- 
tions can be defined and samples drawn from these MIS records, with 
individuals identified for inclusiwi in a study at either entry or termina- 
tion from the program. Onc*e the population — all clients who participated 
in the i^icular programs to be analyzed— is defined, representative 
samples can be assured by using a variety of convenient random selection 
methods. The peculation should include clients enrolled throughout a 
calendar year if the program experiences seasonal fluctuations in out- 
comes, and shcHild proportionately represent all geographic areas from 
which clients are drawn. 

MIS records may be augmented to improve the power of a given study 
to evaluate specific a.spects of program services. In addition, standard 
survey research methods can be employed to perform short-term follow- 
up data collection on program participants or other relevant actors such 
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as employers, treatment facilities, or school administrators. For longer- 
period follow-ups. the problem of noncompletions reduces the value of 
the survey approach. For programs with mobile clientele, establishing 
good locater information cm clients will be critical to the success of 
follow-up survey efforts. The construction of surveys, drawing of 
samples, and conducting of survey interviews arc tasks aboul which 
much is known, making guidance readily available {e.g.. Fink 1985; 
Babbie 1989; Rossi. Wright, and Anderson 1983; Frcy 1983; Dilhnan 
1978). In addhion. the availability of university-based survey research 
centers, as well as private research firms, makes expert guidance readily 
available in most areas. 

Analysis of gross impact data often can be performed on the same 
computers used to store MIS data, using one of die readily available 
statistical analysis packages. Any package that calculates percentage 
distributions, chi square with percentaged tables. analy.sis of variaiM:e 
tests comparing means, and multiple regression is adequate for the needs 
of nearly all gross impact evaluation efforts. 

The Application of Gross Impact Evaluation Concepts and 
MethcMis to the Case Example: JTPA 

The remainder of this chapter illustrates the gross impact analysis 
aK>roach by offering methodological guidance lailorcrd specifically to 
one program, the Job Training Partnership Act. Much of what is s^d here 
applies to any job training program, and with l little translation, to any 
human services program. Conceptualization, design, measurement, and 
analysis issues in gross impact evaluations, however, are illustrated 
within the vocabulary of JTPA services and the JTPA service provider 
system. This section begins with a discussion of the proper uses of 
descriptive outcome analysis, then moves on to differential impact 
analysis and the study of employer outcomes. 

Tiie Uses and Limits of Descriptive Gross Outcomes Analysis 

Descriptive analysis lakes its name from its goal of examining out- 
comes without making causal attributions. Descriptive patterns may be 
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reported on the basis of data covering all participants or estimated from 
a sample of partici]^ts. Where sampling is involv«i. proper procedure 
will generate unbiased estimates of patterns characterizing all partici- 
pants as well as information on how jKxurate those estimates arc. 

Descriptive data are relatively easy to collect and report, but also easy 
to misinterpret. This discussion, therefore, takes two directions: (1) 
identifying ways to make gross imi^ct description most useful to JTPA 
managers, and (2) ictentlfying the major limits on its valid interpretation. 

Avoiding Interpretations that Imply Causality 

As the term description imlicates, the primary limitation on descrip- 
tive data analysis is that it involves mne of the re^arch design or analysis 
techniques designed for explaining causal relaticHiships. Hie major 
reason for this limitation is that descriptive analysis offers no compari- 
sons. For example, if we learn that employers arc highly satisfied, we 
cannot know wtether the reason is the friendly service, the quality of 
participants placed with them, reimbur^mcnt they may have received, 
agenerai tendency to answer positively, or a variety of other p(»sibi]ities. 
We can guess, but the research findings offer no guidance until compari- 
sons arc made — in this example, comparisons bctw(»n employers train- 
ing more- and less-qualified participants, with higher and lower reim- 
bursement levels. 

This limitation does not mean that managers must refrain from 
interpretation. We all interpret the worid daily. It means that managers 
must not assume that the findings themselves imply a particular interpre- 
tation. Thus, the first of the following two statements a JTPA manager 
might make is flatly incorrect, while the second could be correct. 

1 . "We find employers to be highly satisfied with JTPA, proving that 
we are sending them the types of employees they want." 

2 . "We find employers to be highly satisfied with JTPA. In my opinion, 
this is tnie because we are sending them the types of employeco they 
want." 

Statement 2 avoids incorrect causal attributions, while also staling a 
possible interpretation that could be examined through further analysis. 
The value of the descriptive finding is that it identifies the facts the 
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manager may work with afui may attempt to explain. We can Icam how 
satisfied employers are. Tlie limitation is that the fmdings do not 
themselves offer any causal explanation for the level observed 

The most common error in interpnet ing descriptive data on job training 
programs is to assume that outcomes described after the completion of 
a program are cauwd by the program. In the heat of political battle, I may 
say, "Look what our program has accomplished; we have 84 percent 
placement rates!'* In so stating, I may be taking ca*dh for uf»wings in the 
economy, for individuals who recovered from temporary unemploy- 
ment, and for random change, as well as for cases wherc employment was 
produced by the program.' Similariy , if I claim that one service provider 
is **belter * (causes greater success) than another on the basis of descrip- 
tive fmdings, I err by assuming that the difference was produced by 
program services alone, which carmot be cU^monstrated using descriptive 
statistics. Such claims are problematic not only because they sub^t 
to self-scrving interpretations, but also because they often tend to mire 
program development efforts in the most obvious and least useful 
interpretations — some vague improvement in "management," "cieam- 
ing*' efforts, economic shifts, and so on. 

Brmdening the Range of Outcomes Measured 

The most basic way in which descriptive analysis can in.pruve a 
program manager's information base is by taking advantage of its 
flexibility to enlarge the range of outcomes measured. We are too often 
wedded to the narrow range of outcomes readily available from agency 
records or r^uir^ by government performance standards. These meas- 
ures should be included in any descriptive analysis^ but the additicm of 
further outcome measures offers considerable benefits: the identifica- 
tion of unrecognized areas of program quality or problems, and the 
expansion of managerial decisionmaking intoquality -enhancement rather 
than program-compliance only. 

A hierarchy of outcomes may be arranged according to the extent to 
which each is required for a meaningful analysis of JTPA. Some states 
or SDAs may wish to include only a minimal core set of measures, 
making the foltow-up as brief and inexpensive as pc^sible and limiting 
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1* Ri«iir^pi^9n^iop8iftmmmst8^ni$lm^nte 

• En^ioy^ during 1^ after tem^natit^? 

• Earnings dunng 13th wek. 

• Number of wori^. during IS-vimk follow-up peric^. 

2. 0tli8fCimmeei^wi8ip!tettinJTPAmnitite 

• Employn^nt, including: 

— Hours per w^kOTpIo^ at foliow-up. 

— Pre- to postprogram change in hours per week and fwrcent weete employed. 

• Earnings, including: 

— Hourly ifi^ rate at follow-up. 

— Total eamii^ from temtinatton to f oHow-up. 

~ Prepn^ram to postprogram change in wages and earnings. 

« W^aredepemtency, including: 

— Whether rcceh^ng public as^^nce at foliow-up 

— Monthly dollar amount of public assistance at follow-up. 

— Total putriic ^tetance r^^eiv^ between termii^tion and f oLow-up. 

— Preprogram to jMstprogram chan<^ in puWic assistance received. 

» Wither employment is in training-related field. 

• Proportion of the worti utilizii^ skills tmm training. 

• For em|rfoy8r-l»8«3 interventions, retention with tl^t employer. 

4« Hmarssofjftbcittaitty 

• B^its (m^tical. retirement plans: vacations: sick leave). 

• Likelihood of layoffs or reduction in hours. 

• Likelihood of promotion and/or raises. 

S< MtMirBSGlmisi^rUflgtta^mrt&mphvtdMfsl^^ 

• Why termination job was l(^t or left, if applicable 

• Whether participant is ^ing work, and if not. why not 

• Intention to make use of tl^ JTPA intervention {career orientation) 

• Personal f^ltstion of JTPA pn)gram services. 

• (^fsonai evalt^ion of pc^rogram job. 

• Perso(^ comparison of pos^rogran 0 with preprogram job. 
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their analysis options accordingly. Others may wish to mount a more 
comprehensive analysis, once the (tecision is made to expend initial set- 
up costs. Hie maj^ginal increase in cost from inclusion of all six types of 
m^uies discussed below is small, making it logical to measure all. 
Nevertheless, some measures add infonnation without being necessary 
to the research effort. With this distinction in mind, exhibit 3.2 displays 
outcomes in six categories, from highest ( 1 ) to lowest (6) priority. 

The most ba.«c outcomes focus on the explicit JTPA mandate that 
JTPA be coasidered an mvestmenl in individual lives— an investment in 
human capital. As such, it should show returns in higher probability of 
employment, higher earnings, and lower dependence on public assis- 
tance. Three measures are required by the Department of Labor, Beyond 
those, the survey method allows various components of employment and 
earnings, such as hours worked and wage rate, to be specified. 

Although measures indicating skili tranter and utilization arc not 
explicated as outcomes in the legislation, they arc clearly implied. They 
reprc.sent the most direct impact of training-based interventions, and arc 
especially sensitive to program variants, making these outcomes particu- 
larly useful to managers who wish to devclq? their programs based on 
differential impact analysis. They are also particularly useful for descrip- 
tive analysis, because findings indicate, in and of themselves, the extent 
to which outcomes arc being produced via the method presupposed for 
all training programs. 

In addition to wages, various intangible benefits from employment 
and indirect forms of income, such as medical benefits, are important 
aspects of job quality. A prime indicator of probable long-range employ- 
ment success is whether the overall quality of each job pl^s it into the 
category somelunes characterized as the "primary labor market" or into 
the "secondary labw market" (Doeringer and Piore 1971: Vermeulen 
and Hudson-Wilson 1981). Primaiy labor market jobs are relatively 
stable, include gradually impwDving income and beiwfit levels, are 
usually full time, include the possibility of promotion, include fringe 
benefits and are, in general, the types of jobs that can rcascmably b«x)me 
a career. Secondary labor market jobs seldom include benefits, possibil- 
ity of promotion, or a system of pay increments, are often part time, and 
are subject to layoffs. Even where a short-term follow-up shows partici- 
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imnt& i^taining employmait, jcbn in the seccmdary labor market repre- 
sent a poor risk for long-tenr. eiitployment stability. 

Filially, tl» lowest pri(Mity outcomes are measures of participanis* 
satisfaction. They arc lower priority than otter measures because their 
meanmg is less clear, ttey are \ef^ reliably measured than other out- 
comes, ami they have been excluded from most job tiuining legislation 
and evaluation studies. Nevertteless, they are much to be recommeiKied. 
They can offer valuable information to JTPA progmm operators, and 
they are inexpensive to ack) to [mrticipant Interviews. In particular, 
subjective indicators of job adjustment may be extremely important to 
measure in cases wtere tte transition into tte workforce is expected to 
be especially problematic, e.g., in ite ci«e of long-temi welfare recipi- 
ents* Job satisfaction has surprisingly little correspornieiKe to earnings, 
but is ciHisiderably influenced by job quality and skill use, making this 
subj^tive measure useful for asse?^ing quality of program placements. 

Asking Questions 

One important approach to both limits and potentials of descriptive 
analysis is to ask meaningful questions without demanding more com- 
plex comparisons than lUlowed* Some questicms involve no interpreta- 
tion; ttey simply seek baseline descriptive information. Other questions 
may te worded specifically enough that a descriptive answer will assist 
tte analyst in developing or confirming explanations. The following 
questions illustrate this point: 

1 • Does it appear that program goals are teing met? If I know roughly 
which levels of program outcomes are expected, measuring out- 
comes lets me know wtelher I am in condition red, yellow, or green. 
Descriptive levels do not tell why outcomes are higher or lower 
than exp^ted, or wtetter my program itself has miK:h to do with 
producing thc^ outcomes. However, ttey tell me whether I need to 
look for factors creating low outcon^s, wtetter high or low outcome 
levels arc concentrated in pmticular program activities, and wtether 
my organization is in tetter shape with regard to some outcomes than 
otters. That is, descriptions of outcome levels can let managers 
know wtetter to worry, and which program areas to worry about 
most actively. 
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For jiome po^tprograni outcomes^ — those mandated as postpro- 
gram perfcHTnance standards— clear expectations will be estab- 
lislHxl, When expectations are unclear, descriptive measures can 
twlp establish reasonable Niseiine expectations. TTicm would cc«isti- 
tule first approximations that might be improved upon in subsequrat 

2. Does any service iWDvider appear worth learning more about? One 
particularly useful application of descriptive an^ysis as a first 
approximation is the comparison (fSDAs or suhcoMracting service 
providers. Such comparisons idiould be interpreted with great care* 
since agency performance levels are influemred by fetors over 
which program operators have no omtroU such as tte local ecwiomy , 
or may result from policies not intended by the act, such as increasing 
performance rates by serving those with Ici^t need. Descriptive 
differences point out where further investigation might be most 
useful, hciping to pose questions correctly rather than answering 
them, 

3. Is ihcK any apparent change over time? Dcscrii«i vc cHitCimie figures 
kept over lime— «ach month for example — can be to form a 
baseline series indicating stability or change in services provided 
and program outcomes. Such a '*time series'" can sometimes alert 
managers to UTOxpected changes* It can also provide a relatively 
inexpensive first approximation of the effects of major program 
changes made during the time series. 

Investigating Specie FrapasUions 

One major strategy of multivariate analysis is to test a particular 
interpretation by seeing whether competing explanations for the d>- 
scrved findings can be eliminated. This tactic is not available for 
(tescripiive analysis* However, the same geiwral strategy may be fol- 
lowed by posing questions thoughtfully and specifically enough to 
logically reduce the range of findings that would be consistent with the 
particular explanation proposal. 

There is little value in asking bn^d questions such as: does on-ihe-job 
training (OJT) prodiKre more placements than classrcwm training <CT}? 
Too many different interpretations could reasonably explain either 
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|x>sitive or negaiive findings. However, specific profH>sitions direct 
expeclalions to only a few findings. // the expected finding cxxurs, then 
we have greater faith in the correctness of the pro{X)sition guiding the 
analysis*' 

For example, if I identify some JTPA prognmi activities as skill train- 
ing programs, I will expect that a disproportionaie number of iK>slpm- 
gnim job placements will be in the skill area. I have no a priori way to 
set expected levels, but descriptive findings are nevertheless interpret- 
able. If only 2 percent of workers in my area are cashiers, and only 6 
percent of my CF participEmts have previous experience as cashiers, then 
a finding that 65 penrent of employed graduates from my easier training 
class are cashiers suggests that the program is working in the way I 
envisicmed. This does not indicate how well the program works, only that 
my proposed explanation about the way if works is supported. 

Another example involves the question: what accounts for nonreten- 
tion of jobs held at JTPA termination? Several specific propositions arc 
easy to imagine, each suggesting its own specific measures. For example, 
if JTPA participants l^k the ability to leam complex skills, instances of 
nonretention should occur most often when the training or the job 
involved complex skills or wherc the p^micipant's preprogram skills 
were weakest. Similarly, employers should often report that the partici- 
pant was unable to perform complex tasks. If these variables arc 
measured along with others indicating alternative explanations, manag- 
ers can assess which explanations account best for the patterns observed. 

As a final example, one may argue that JTPA should move participants 
into primary labor market positions (Taggart 198 1 ). One c^uld examine 
the degree to which this incurs by measuring qualities of ;x>stprogr^ 
jobs that define the primary labor market, as shown in exhibit 3,2. 
Findings would not indicate the degree to which JTPA treatment caused 
the job quality mix observed, but they would recommend greater or 
lesser concern about program quality, depending on the number of jobs 
exhibiting the desired qualities. 

Perce/^ons Held by Employers and Participants 

Some questions are inherently descriptive. If I wonder what impor- 
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tance employers place on various qualities of individuals Ihey hire, I can 
ask them lo tell me. Although it is always possible for data on such 
perceptions to be limited by incorrect self-knowledge or by misleading 
responses, these perceptions are appropriately interpreted in their <te- 
scriptive form. The same is true of participants* job satisfaction or other 
participant perceptions in which JTPA managers may have interest. 
Similarly, employers' satisfaction with JTPA and their perceptions of the 
costs and benefits of participating in OJT or Work Experience may be 
taken at face value, as long as one recognizes that the infonnation 
indicates no more than perception, and that perceptions do not necessar- 
ily reflect program impact. 

Illiistrations of Informative Descriptive Analyses 

Several illustratiwLs of descriptive findings are rejxjrted here, all taken 
fjx>m one statewide study of CETA OJT conducted in Washington Smte 
(Simpson 1984a). The findings displayed in exhibit 3.3 illustnUe that 
measuring training-related employment as well as overall employment 
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at follow-up t^lps avoid jumping to an erroneous conclusicm. If only the 
percent emplo)^ or not employed at follow-up wei^ displayed* we 
vmvAA ob$CTye the expected pattern: AFDC lecipienu^ arc employed at a 
rate about 9 percent below the rate for iwn- AFDC particii^nts. We might, 
therefore, be led to conclude that AFDC rwripients are less job-ready or 
less personally stable, ami tfKjrefore fail more often than others to retain 
their OJT jobs. A program manager might ccmsicter imposing more 
ccHinseling on AFDC participants offering more support services during 
OJT contracts, or placing fewer AFDC nxipients in OJT, aldiCHigh it is 
widely known to produce tl^ highest postprogram placement rates. 

However, when the outcome is presented with a slight increase in 
specificity, these interpretations no longer appear logical* AFDC and 
mm-AFDC participants retiun their OJT jobs at equal rates. They also 
move to other, training-related jote at equal rates. The entire differential 
in employment is produced by tlK? fact that among the 60 percent of 
participants who did not remain with their OJT employers or in the field 
of their OJT positions, non^AFDC recipients were two-thirds more likely 
to find work outside the OJT field. Now the most likely interpretation is 
that AFDC recipients are much less able to locate jobs without the 
assistance of the job training program, as shown by their relative lack of 
success once they leave the positions into which they were leveraged by 
OJT subsidies. However, once the program assisted their job entry, they 
retained tteir positions as often as their more employable colleagues. 
This means that the OJT program is doing well at equalizing the chances 
of AFDC people in the short mn, tnit it also means that the OJT program 
effects are not carrying over to later job search success/ 

Exhibit 3.4 illustrates the value of an extremely basic analysis of 
follow-up data, A group of OJT participants is followed from the 
beginning of their OJT contracts through contract compIeti3n ami nine 
months beyond. The percent of the initial group who remain employed 
with their initial OJT employer has been calculated at several points 
during the OJT contracts, and monthly after termination. The findings 
gr^ed in exhibit 3.4 fall into three segments, so clear as to be quite 
valuable in their descriptive state. 

During the OJT contract, some gradual attrition occurs, so that on the 
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date of coniract completion, only 83 percent of the original placements 
remain/ Then, a full 30 pcrccnl of all OJT jobs arc lost or left in the month 
following the termination of the contract. Following this stage, attrition 
once again becomes gradual, with another 21 percent of jobs being left 
over a ninc-miMith period. When the findings graphed here are combined 
with an official "entered employment at termination** rate of 78 percent, 
we also learn that the gnrat majority of jobs lost during the first month 
following the OJT contract were counted as program successes. Despite 
their simplicity, these findings speak unambiguously alx)ul the value of 
postpnagram follow-ups and the way in which the OJT system at that time 
and place was working. 

The final illustration of descriptive findings offered here comes from 
a small rccent survey of OJT employers in one SDA, The figures hertr 
suffer a relatively large eiror margin since they are based on only 78 
interviews. However, employer resptmscs were so extreme thai the small 
numbers cannot obscure the basic thrust. Exhibit 3.5 reports one set of 
employer perceptions relevant to an interpretation of employer cost or 
benefit from participating in the JTPA OJT program. Aside frt>m the 
wage subsidy OJT employers receive, the greatest cost or benefit they 
receive fnm participation in OJT is the work produced by the participant. 
If the participant is less skilled, slower to learn, less productive, or kss 
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tractable than norma! non-OJT hires, the placement represents a cost. 
Indeed, the wage subsidy is supposed to offset such co^ts. R)r this reason, 
employers were asked to compare their OJT participants, at the point of 
their initial hire, with the typical non-OJT hire for the same position. 

The results sbt»wn in exhibit 3.5 make clear that in this SDA. the great 
majority of employers perceive that they have benefited from hiring an 
OJT participant. Over 80 percent isay their OJT hire is easier to supervi.se 
than other hires, and over W) percent say the OJT hire is a more productive 
worker than non-OJT hires. The same pattern holds for all the specific 
ratings save one; 3 1 percent of employers report their OJT participant 
needed grealer-ihan-average training, while only 1 1 percent say they 
needed less. Even here, in the area assumed by definition to represent a 
cost to OJT employers, the majority say OJT and non-OJI hires are 
identical. 

This descriptive finding is chosen to illustrate both the value and the 
limits of descriptive findings. Program planners can feel assured of the 
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siatisfaction with which their OJT program is being greeted, can use these 
rtmiis in marketing OJT to other employers, and can rest a^urcd that 
their assessment and assignment system is locating successful workers. 
Yet, they cannot kirow how much of this employer satisfiwrtion stems 
from excellent matching of particular employer needs and particular 
participant strengths and weaki^sses, how much from the general 
employment maturity, which is by policy required of all OJT partici- 
pants, and how mttch from "over-selection** of the most employable 
individuals to be placed in Oil Many policy implications depend on 
these alternatives, although some implications are clear in any event* 
such as the feasibility uf reducing wage subsidy without inducing 
employer perceptions that OJT is ick> costly. 

Minimal Research Design for a Parttcifmnt Folh>w-Up Analysts 

Whetter analysis will rcmain descriptive or move on to differential 
impact, certain minimal research design requirements guide the proper 
collection of survey data. A large amount of literature is available on 
survey research methods (see especially Rossi, Wright, and Anderson 
1983; Dillman 1978). In this chapter, a set of topics that must be 
addressed by any client follow-up survey is listed with only brief 
comments on advisable design decisions for JTPA* A more extensive 
treatment of the design issues facing JTPA evaluations is found in 
Simpson (1986). 

Identtfying the Population to Be Analyzed 

The first step in designing either a descriptive outcomes analysis or a 
differential impact analysis is deciding which >^t of participants to 
include — ^Ihal is, how to define the pi>pidatUm under study, the popula- 
tion to which conclusions will be gei^ralized. In JTPA, the fin;l such 
decision involves which authorizing titles are to be included. This choice 
depends primarily on managers' goals for the analysis. In acklition, 
attention must be paid to the comparability of measures across title and 
to whether the programs operated under different titles arc comparable 
enough to be combined meaningfully. 
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Tte second decision regarding which fK>pu!ation to study is whether 
the populaiion should include the following: all of those found to be 
eligible for JTPA services, all of those who were enrolled in JTPA, all 
statuses at termination* or only participants who were employed at 
temiinaticni. Studying all eligibles would be required for a full analysis 
of selection into the program. However, for a ctescriptive analysis of 
gross outcomes or a diffenential imfxict analysis of the effects of program 
scrviw: or implementation variants, only participants receiving services 
need be included. 

There is sometimes a temptation to reduce data collection costs by 
including in the study population only individuals who terminated with 
employment. However, there are sevei^I reasons why s^oss impact 
analysis desi};ns should include all iermination statuses in Ihc popula- 
tion tc be studied.** The only outcome that can be me<^ured with a 
{K>pulation limited to particifHmts who were employed at termination is 
"retention of the termination job/' Estimates of other standard mitcome 
measurcs^-average wage, proportion employed, etc.— would be tedly 
inflated by excluding the group least likely to be employed at follow- 
up— those unemployed at termination* On the other hand, toestimate this 
group*s follow-up employment and earnings at zcrv (their status at 
termination) would underestimate pn^gram success by ignoring delayed 
employment. Further, including all termination statuses allows the 
evaluation to examine why some individuals gain less than others from 
the program, insures comparability with cost data^ and guards against 
differences in service providers* methods of defining termination status. 

lidding Whether Data Collection Should Be Longitudinal 

In order to analyze change in employment, earnings, life satisfaction, 
and the like, the same individuals must be measured before and after the 
program intervention. In some situations, program eligibility data par- 
tially satisfy this need by detailing work history as well as individual 
background characteristics. When these data constitute adequate prepro- 
gram treasures, an evaluation may be "added on** efficiently by identi- 
fying a sample ai the point of program termination and measuring 
postpmgram outcomes parallel to the preprogram measures already 
available. 

t r . ^ 
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A wide range of prcpn>gram data may be inf«miative ctmceming client 
characteristics. The imjxirtanl issue regarding the analysis of change is 
that the preprogram measures be similar enough to postprogram meas- 
ures to be comparable. If the range of program outcomes is widened, 
additional preprogram measures may be required. However, pre- and 
postprogram mea.surcs need not be identical, as discussed later in this 
chapter. 

Determining the Durban of the Study 

While it would be convenient to ccMiccnirate data collection into a few 
months, this shortcut endangers the validity of the research, introducing 
known biases and others less easy to identify. The population to be 
studied should, therefore, be defined to include uil enroilees or tcrminees 
throughout the full year. 

This definition prevents bias due to seasonal variations in the labor 
market. Similarly, in classroom training, some institutions tend to end 
courses during particular months, so that the proportion of tcmiinees who 
are program completers vs. those who are dropouts varies monthly. 
Third, different service providers develop different policies concerning 
when to commit their funds, in total or for particular services, and how 
long to hold unsuccessful participants before terminating them wten 
required to by the expiration of agency contract periods. All these factors 
produce seasonal differences in the likelihood of program success. 

Defining the Size of the Study Sample 

Once a {xjpulation is selected for analysis, the question becomes how 
large a sample should he in order for calculations to estimate accurately 
tlie patterns within the entire participant population. Assuming a repre- 
sentative sample, the primary determinant of error margin (i.e.. of the 
accuracy of conclusions) is the number of cases uptm which estimates are 
based. A conclusion that one program variant has 10 percent higher 
retention rate than another means little if the margin of ennor for that 
estimate is 20 percent. Therefore, the first decision must be how many 
ca.ses are needed in order to generate a level of error acceptable to those 
who will u.se the analysis results. 
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One essential reason that survey a^search has become such a widely 
used method is thai the accuracy of eslimaies rises rapidly as we move 
from very small samples to samples of modest size: yet, samples of 
modest size arc nearly as accurate as very large samples, TTiis occurs 
because error decreases as a function of the square wol of the sjmiple size. 
More precisely, the estimated era^r associated with any measure depends 
on the srundurd error of that measure. For random samples, the standaixi 
error equals the standard deviation of the measure divided by the square 
root of the sample size {i,c,,se = sd stfrrS). Thus, if an income measure 
has a standard deviation of $4i)(Xl the standard error is about $8(X) with 
a sample of 25, $4(K) with a sample of mi $2(K) with a sample of 40( , 
and $100 with a siunple of 1 .frfX). 

One can see the danger of rclying on a veiy small sample. However, it 
is equally evident that the marginal improvement from each increase in 
sample size is quickly reduced as sample size becomes lai^cr. In the 
example above, adding 375 to a sample of 25 reduces error by $6{X), from 
$800 to $2{X). However, another 1,2(X) cases would be required to trim 
a further SI 00 off the standard emir. 

This phenomenon explains why many slate and l(Kal surveys with 
limited funds choose sample sizes in the range of 350-500. Efficiency 
(accuracy gained per Increase in data collection cost ) rises rapidly below 
that level, but more slowly aflerwani. In addition, when percentages 
(e.g., percent "yes") are reported, samples in this size range produce eiror 
margins at or under 5 percent, a round number and error margin typically 
satisfactory for most purposes. However sample si/e decisions should 
always be made after analysis goals are clearly established. Planners 
will be well-advised to consult one of several thorough texts on sampling 
(e.g., Kish 1965: Sudman 1976) or to employ a sampling specialist in 
casv3 where sampling appeani problematic or in order to determine the 
most cosl-efllcient sample.^ 

One additional consideration must be included in planning sample 
size: not all members of the initial sample selected will be contacted. 
TlK^rcfore,thc number of specific individuals .selected for inclusion in the 
sample must be greater than the number of completed interviews desired. 
The number to be selected is calculated by dividing the desired number 
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of compleled inicrviews by the planned survey cimiplclion raic. For 
example, to complete 400 interviews at a completion rate of 70 peavnl 
would require that 57 1 names be identified in il^ initial sample (400 0.7). 

Stratifymg Sam/ries 

Populations are sometimes divided into subgn^ups, or strata, each of 
which is sampled separately. Strata may be sampled in proportion to their 
numbers in the population, ordtspix>portionaiely. Although some believe 
that samples must 1:^ proportionately stratified to insure the proj^r 
number of members with various background characteristics, this belief 
is in error. Proper sampling procedui^rs insuii? a representative sample. 
No reason exists to comidcv proportionate stratification in gross impact 
analysi.s. 

There are several conditions under which disproportionately stratified 
samples are sometimes recommended (Sudman 1976). Only one of ihew 
is applicable to gross impact analysis, but that one reason is central to 
statewide differential impact analysis as well as to analyses of subcon- 
tractor performance within large SDAs. The need for disproportionate 
stratification of gross impact samples ari.ses when the analyst's emphasis 
is on comparing or reliably chaiBcterizing subiH^pulations rather than on 
characleri/jng the entire population of participants. This occurs in the 
case of JTPA postprogram performance standards, where welfare recipi- 
ents arc trcated as a sepai^le stratum. In addition, statewide analysis 
aimed toward comparisons among SDAs, or SDA-level analysis com- 
paring service providers, should consider stratifying to insure reliable 
characterization of smaller units. For such comparisons, the operative 
issue is not total sample size, but sample size for each subunil being 
compared with any other. 

Identifying Members of the Sample 

The sample of participants who are interviewed must be representative 
of the population being studied: i,e., each element of the population must 
have an equal chance to be included in the sample. None of the claims for 
sample efficiency or reliability holds w hen samples are not representa- 
tive. Sample selection procedures must guardntce equal probability of 
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inclusion, climinaling any purposeful or accidental selection. Tte only 
way to guaranty equal probability of iiKlunion is to select from the 
population into the ^mple at ratKlom. The classical approach is to select 
each imlividual rit>m an ordered list» using a table of random numbers. 
Two, more ccMivenient methods, however, are equally valid. 

First, the last ihrre digits of fmicipanis* sixrial security numbers are 
random with respect to any meaningful characteristic of individuals. 
Therefore, sample members may h: selected by icfentifying a range of 
three-digit numbers that would produce the required sample size, and 
including all particijwils with numbers falling in that range. If, for 
example, 25 percent of the names are to be sampled, the lower end of tte 
range is chosen at random, and the upper is set at 250 higher. The second 
method is systematic sanipiing based on a random start. If 25 percent of 
the population is to be sampled, a list of names, typically a computer file* 
is prepared. One of the first four is chosen at random, and then every 
name is included in the sample, 
jlicjpants may be selected into the sample at citY^r proffrani entry or 
termination, as long as the full iK)pulation of participants is available for 
the sample. From the practical research administration viewpoint, it is 
often preferable to identify the sample at termination. However, if the 
data collection plan requires the addition of individual treatment meas- 
ures throughout the program, it would be most efficient to identify 
participants for inclusion in the analysis upon entry. At that point, they 
could be specially tagged for collection of individual treatment data and 
inclusion in the postprogram follow-up. 

Establishing a FoUow-Vp Period for 
Measuring Postprogtram Outcomes 

In addition to outcomes measured at program termination, postpro- 
gram outcomes are especially valuable tor assessing program quality* In 
the case of JTPA, a three-month follow-up is required, making that 
period the obvious choice for a fii^t, and pethaps only, follow-up. The 
majority of any survey costs occurs before the first question is asked — 
recording locater infonnalion, identifying a sample, keeping records on 
that sample, tracking hard-lo-locate former participants, hiring and 
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training inlerviewers. setting up interview phone banks, making multiple 
calls to locate the participant, and introducing the puipose of the calK 
Thcrcforc, the most efficient design is lo add questions to a pre-existing 
survey. 

In addition, the issue of seleciing a follow-up pericxl should be 
examined on its own merits. The major question is whelhei three months 
is too short a follow-up lag pericni. Factors weigh on both sides of the 
question* 

There is value in extending the follow-up time period. Recent studies 
testing how well various follow-up measures predict long-term net 
impact of JTPA find three-month follow-ups much stronger than termi- 
nation data alone, six-monih follow-ups stronger than those at three 
months, and nine-month periixls stronger than those at six months 
(Gcraci 1 984; Zomitsky et al. 1 985 ). While the gain from each additional 
delay is smaller than the one before, each does offer improved validity. 

Costs are also involved in extending the follow-up period. Follow-up 
surveys are subject to serious sample atmtion if the first or only 
imerviews are conducted very long after termination. Since sample 
attrition introduces unknown biases, it is preferable to conduct shorter- 
term follow-ups and achieve higher completion rates. In addition, a 
three-month follow-up is minimally acceptable. That delay is long 
enough to allow the mte of employment after classnx>m training to 
stabilize. It is also long enough for OJT placement to stabilize after the 
postcontiBct drop-off, even where 3()-day delayed performance pay- 
ments may delay thai drop-off. Three months is also long enough to 
eliminate most inconsistencies introduced by the tendencies of some 
service providers to make more extensive use than others of the "admin- 
istrative hold for joh search assistance*' category following the program. 

Given the costs and inefficiencies of long-term follow-up surveys^ 
states or SDAs planning to do longer-term follow-up may wish to 
consider using unemployment insurance (UI) wage records if they are 
available. Once access to the UI data base is established, a one- or even 
twcvyear follow-up is as easy to perform as a six-month follow-up. (UI 
system use is detailed by Johnson and Stromsdorfer in chapter 2 of this 
volume.) One factor, however, limits the usefulness of these data as a 
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gnm iinpacl mca*iunr: U! data cover only individuals who maintain 
residence within the stale. In \bc net impact approach, movement out of 
stale is assumed to be equivalent for treated and untreated groups. 
However, gross outcomes are measured only for particij^ts. msdcing 
movement out of state a serious problem. One cannot determine whether 
a record of zero earnings represents continuous unemployment or move- 
ment out of the state. Use of this approach is recommended only if a 
separate tracking effort — to estimate the proportion whi> moved out of 
the UI rcpc^rting arca — is mountc*d for thi>st individuals with zero Ul 
income. That estimate could then be used to adjust estimated job 
retention rates. 

Choosing the Data Collection Method 

Gross impact anal>sis involves data collection ihn>ugh follow-uo 
surveys of participants and employers. The rapid expansion of the survey 
nrsearch industry has N^cn accompanied by a growing literature on how 
to conduct surveys, the strengths and weaknesses of surveys, and the 
relat've strengths and weaknesses of in-person interviews, mail ques- 
tionnaires, and phone interviews (e.g.. Dillman 1 978; Ri>ssi. Wright, and 
Anden^on 1983), Survey i^seaah technology will not be detailed here; 
it will suffice lo make the following summary claims: 

1 . Correctly conducted, surveys have proven highK mliable. 

2. SiiTveys suffer tnuch less resptiase bias than once feared, as long as 
the respondent believes the interviews arc conducted by a neutral 
party.'* and interviewers converse in a natural style (Bradbum 1983). 

3. Bias from nonresponse can be problematic, but can be guarded 
against by achieving relatively high response rates luid either insur- 
ing relatively equal res}X)nse rates from all key subgroups or statis- 
tically adjusting subgroup response mtcs during analysis. 

4. Given the cost of in-person interview: . the high nonit^sponse rate 
and possible educational bias in respo, vith mail questionnaires, 
phone surveys arc usually recommended for program evaluation. 

Conclusions Concerning l>esign 
When the research design decisions just discussed have been made, the 
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basic structure of liie evaluation effort is in place. Tliese issues have been 
touched on only briefly, as they arc so basic. However, detailed treat- 
ments arc available In various standard texts (e,g., Rossi and Freeman 
1982), making further comment here unnecessary. 

Differential Imp^t Analysts: 

Estimating Influences on Prt^ram Outcomes 

The primary goal of differcntial impact analysis is to reliably describe 
diflerences in p(^tprognmi outcomes i^ross program services and fomis 
of service delivery implementation, so as to identify the prof^Ne cai4scs 
of those differences. In nonexperimental research, identifying causal 
relationships is problematic. However, quasi-experimental research 
designs, such as tfwse rccomrr^ncted by Campbell and Stanley (1966), 
can considerably increase our confidence in having reliably identified 
the major causes of differences we observe. (See also Campbell and 
Cook 1979: CafK)raso and Roos 1973.) For purposes of program 
development, managers will wish to gain information about the differen- 
tial effectiveness of program variants — options available to managers in 
the services assigned to individuals and in fomis of program implemen- 
tation. These variants may be altered on tl^ basis of evaluation findings 
in order to nnprove program effectivetiess. 

Formal, quantitative differential impact analysis is a relatively under- 
developed field, as is well-illustrated in Boms's (1979) program evalu- 
ation primer. After listing 44 specific participant characteristics lenown 
to affect labor market success, inrr*^ question of **program 

component independent variables/' *ne-paragraph discussion of 
this topic begins by saying thai *\ . Id be extremely useful in 
modifying existing programs and in th*. . jming of new programs to 
know which of the (ptDgram) components is most effective for various 
types of participmits** (Bofus 1 979, p. 70). Two measures are suggested: 
program length, "... and, if possible, a measure of quality.** A review of 
previous studies using a differentiaJ impact analysis approach to evalu- 
ating job training programs is included in Simpson (1986). 

Identifying probable causal connections is valuable to program man- 
agers because changing a factor that has a causal influence on program 
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oufcomcH is likely to change the level of those outcomes. While formal 
causal mix^Hng may seem the domwn of esoteric social scieiKe, it is 
ess«:ntial to research on the basis of which investment dccisiims may be 
made. Let us suppo^^e that a weak analysis confirms higher success rates 
among individuals who receive shorter intake pn>ccdures and concludes 
that brief intake cau?^ improvoi program performance. Let us further 
assume that a more solid causal analysis would have learned that tl^ most 
highly employable fmrlicipants entering the system were given brief 
intake becau.^ they had little need and that it was their cmployabiliiy 
rather than the intake that affected their postpn)gram sucx*css. Ironically, 
if an SDA were to base decisions on such weak research and overhaul its 
intake system to offer only short intake, its performance would not 
improve and might deteriorate, because those jwtlcipants who needed, 
and previously received, the longest intake no longer have that opportunity. 

The goal of each of the steps involved in differential impact analysis 
is to increase our confidence that we have identified those program 
variants that do have a direct intluence on program outcomes and are 
therefore useful to program managers in improving their programs. This 
chapter can only summarize some major characteristics of nonexperi- 
mental research designed to increase the analyst's ability to identify 
causal relationships. There are also useful references available on this 
complex subject (e.g., Blalwk 1%4; 1985). 

Research into causal relationships begias with comparisons. To deter- 
mine whctlKr program option A is better than option fi, one must idem i fy 
a criterion of comparison (e.g., job relent ion ) and compare options A and 
B on that dimem;ion. Options could basic program activities, such as 
OJT, c!.tssn>om training, or work exf^rience, or optional variants within 
the same ?»cti vity , such as OJT assignments developed by the p^icipant, 
developed by the service provider, or initiated by an employer. These 
comparisons should be selected so that a causal interpretation is reason- 
able. This is where past research findings, economic theory, and manag- 
ers* knowledge of programs come into play. If answering a question in 
causal terms would fiy in the face of logic or of established information, 
the question probably j^iould not be posed as part of a differential im|xict 
analysis. 
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In addition, to convincingly establish ihai a relationship is causaK 
findings fn>ni our comparisons must hold up after competing explana- 
tions have been eliminated. Each lime we identify a plausible comfHrting 
explanation, test it, and find that it does not explain away the difference 
between options A and we increase our contldence in the causal 
association beiv^cen program variant A/B and the outcome in question. 
For example, our confidence in finding that public classawm trainers 
perform Knier than proprietary trainers (or the reverse) would be 
increased by learning that the difTerence in ixrrformancc could not be 
accounted for by differences in participants" literacy skills or prior job 
experience, by differences in the fields for which each type of school 
trained participants, etc. 

The goal of quasi -experimental research is to eliminate the effects of 
all important measurable alternative explanations. That goal is never 
reached, but we can eliminate many im}:M>rtant alternative explanations. 
These include both factors of interest to the analyst, such as other 
program variants confounded with the one being tested, and control 
variables, such as agr or gender, known lo afTect the outcome in 
question. 

Classical experiments attempt to eliminate competing explanations by 
controlling variants other than the AlB comparison of interest and by 
ramlomly assigning individuals to variants A and hoping thereby to 
prixluce groups equivalent in ail regards except for the variant under 
study. Quasi-experimental research (K:cUi*s in settings that allow neitter 
the control of variants other than those diirctly under study nor the 
ramlom assignment of participants to pmgmm variants. Instead, multi- 
variate statistical techniques are used to determine whether alternative 
explanations are able to undetmine our confidence in findings. 

The primary strategy of differential impact analysis is to lailize each 
program variant as a comparison group for each other variant, Exc^ept 
wtere program variants are too highly correlated with each olher or with 
participant background characteristics, multivariate anahsis can esti- 
mate the unique effects of each. This .same approach has guided recent 
re»»earch on the impact of college (Aslin 1977). In the case of college 
impact, no untreated comparison group exi.sts, making net impacn studies 



Hvaiuatinft Onyss ftograni Outcomes 165 

impossible. However, comp-^risons among colleges, w iih each acting ;ts 
comjMirison gmup for the other* are pK>ssible given careful measurement 
of diffenrntial selection into each college. ITie same logic applies to 
dincrential impact analysis, when? the decision is made to structuii: 
research that cannot ask net impact questions but which can. nonetht-less. 
reliably compare difTerent treatments and ircaimeni contexts. 

Combating Bias That Threatens the 
Validity of Differential Impact Analy?Ms 

The comparisons demanded by tlK analysis goitls and the data collec- 
tion method— surveys in this case — determine the major threats to the 
validity of differential impact analysis. These risks are summarized 
below. Each is a source of hUiw as opjx>sed to ramiom ermr The term 
bias refers to error that consistently misdinxis reseaa-h n?sul!s. Like a 
compass with a metal object neariiy, readiiigs from the analysis are 
distorted in a consistent direction. To correct the Hndings, or** must 
remove the object or adjust for i! > influence. Random error differs fr^m 
bias in that it takes no panicular direction. Random error can be as serious 
as bias if it is large. However, techniques for minimizing random error 
arc well-developed in survey research (i.e., careful measurement tech- 
niques and properly constituted samples). 

Some types of bias can also be dealt with through standard survey 
researeh techniques. These include bias from censored samples, nunre- 
sponse bias, and response bias. TTiey can be prevented by selecting 
samples correctly, achieving high response rates across ail major groups 
in the sample, and wording quesiions properly. The two types of bias 
defined in exhibit 3.6, sdedion bias and bias from confounded program 
vuhmm. are* combated during multivariate analysis, lliis means that 
descriptive gross outcomes analysis, which does not employ multivari- 
ate techniques, is always subject to .;t»rious bias. Differential impact 
analysis is able to adduce, but not eliminate, these biases during analysis. 

Bias is adduced when equations include measurers indicating selectiv- 
ity and program variants that overlap with the program variant being 
tested. However, many selection biases are unknown or cannot be 
measured, making statistical adjustments dilTicull. Therefore*, selection 
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Two Ttfp^ a Bbm ^^ity nuMaimrtte 



Wm partici|:^nts who select or are se^Awi into differ program varmnts ditter in ways ttmt attect 
program outcome, observed outcome diffefene^ between program variants ojyld be produced 
eittert^ program quainies or t^part»:i|^ tf{mrtict(^char^:terte^ are not taken 

into ac^nt ^rruttes of program tmpmn m\\ be biased. For example, pr^mtgram ^ioyimnt 
expertence influences pc^^rogram empto^fty and may ateo infli^nce which program i^rvN^ is 
assigned. W wnptoyment history not m^tsur^. the aratyst cannot adjust for its impjKJt on 
pc^fCHjram outctmms. prcHluang t^sed estimate of the ditf erentm) imp^ of $$rym assignmem. 
Sif^ many such differm^ n^ ex^ but not be nmasurat^ {e.g.. sub^cbve motfvahon), some 
degree of sei^^ bias is ah^ fK^nt m differential fmp^ analysis. 

When two prc^ram variants are correlate with each other, tt^ uniqi^ effete of each can te 
estimate if both are included in the same equation. Howe>« ^ if one is omitfed, then the estimate tor 
the indud(^ ^»nt wiH ^^b the effed of tt^ omitt^ one. biasing nnicli^ns in that direction. For 
example, tf in a particular service delh^ry system, service provijfers who exercise especially careful 
quality control over empto^r sites acceptable for OJT assignments ate) conduct more eWtorate intake 
a^ssn^t of job maturity prkix to OJT aKignm©*. and if both of ftim imprwa OJT jK)Stprogram 
outcomes, tl^n both must be mcluded in the anat^. or the estimated impact of either om alone will 
be inflated. 



bias is the inosf serious anaiyitcai problem as well as 1\k most diHlcuIl to 
diagnose. 



Minimizing Bias from Nonramhm Selection 

Reducing the effects of selection bias follows the general logic of 
causal analysis. Each source of selection bias is an alternative explana- 
tion that can be countered only by inclusion in multivariate equations of 
variables which identify the selection proc^e.ss. The discussion below 
identifies four major sources of selection bias. For each, the aspects of 
differential impact analysis most likely to be affected and the measure- 
ment strategies most able to minimize the bias are indicated. 



Sources of Selection Bias 
I . L^f^aUy eli}(ihle individuals may or may not apply to JTPA because 
of difTerenccs in information available. jH^rsonality or motivational 
differences or geographical differences in services available. Tliis se- 
lection prc^ess is critical for net impact studies, but seldom biases 
differential impact analysis, which involve only comfwisons among 
individuals already enrolled in JTPA. However, if this type of bias 
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differs across SDAs, then statewide differentia! impact comparisonH 
among SDAs will be affected. 

Linle protection from this type of bias is available to analysis that 
docs not include an untreated comparison gnjup. However. SDA- 
levcl measures of program availability and participant measures of 
motivation for applying to JTPA may help assess possible differ- 
ences between SDAs. In addition, standanl demographic back- 
ground characteristics may be correlated with motivational charac- 
tensiics. allowing their inclusion in differential impact analysis to 
act as a partial proxy for direct measures of motivation. 
Panicipants may or may nat he enmiled into JTPA after eligibility 
is determined. If the reasons arc coirclated with program outcomes 
bias will result. The source of this type of selection may be program' 
policies and practices such as targeting, a participant's choice after 
learning of program options, or failure to locate a program placement 
of the type decided on for that participant. 

Mea.suring the st>urcc and nature of the selection is the approp- te 
tool for reducing bias from these sources. It is possible to measure 
implementation policies such as targeting, which are intended to 
determine which eligible individuals are enrolled. In addition, 
measures of participants' demographic and work history character- 
istics may act as a proxy for agency selection or may indicate which 
participants best fit fie agency's desired targets. Beyond that, the key 
measures of agency .selection involve the proportion of eligibles for 
each provider who fail to enroll and the reasons why they have made 
that choice. Given some JTPA manager^;' reported emphasis on 

cnrolhngihe most qualified participants;, statewide differential impact 
analysis will be well-advised to include agency-level measures of 
intended and, where possible, actual selectivity by ser\ ice providers. 
Participants may request a particular treatment aside from the 
decision to enroll in JTPA. If the rea«>ns for that request also predict 
that participant's likely program outcome, this self-selection can 
bias estimates of how program activity affects outcomes. This is 
especially likely to occur where employee select desirable job 
applicants and then send them to JTPA to request OJT enrollment 
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or where schools send their best students to JTPA for support to 
complete a program. Participant ivquests arc most likely to involve 
a basic program activity, or a particular school or employer. Analysis 
comparing these most basic treatment variants is. therefore, the most 
likely to suffer from this source of bias. 

The best protection against bias fn>m self-selected treatment is 
measuring participants* route into JTPA: whether they requested 
particular services, and if so, which ones and why. In particular, the 
mute from employer or school to JTPA should be identified, since it 
involves a type of self-sclcclion most likely to affect postprogram 
success. One could also measure the degree to which particular 
service prtwiders control the assignment to treatment vs. allowing 
participants to elect their own treatment. 
4. Partidpants may also hi' assigm d to particular trvatmaits by pro- 
gram manafivrs If treatment -4 rather than treatment B is assigned on 
the basis of factors that also influence program outcome, selection 
bias is present. 

This source of bias has ptrtentially pervasive effects on differential 
impact analysis bccau.se rational service provider policy offers the 
most intensive services to those with the greatest need. That is. many 
JTPA services are intentionally compen.satory . Sifure "greatest need" 
often translates to "lea.st employable." the selection of services on the 
basis of need can bias estimates of ircatmcnl impacts on employment 
outcomes in studies that do not include measures of need in the 
analysis. To identify compensatory effects of treatment, one must 
have measures of both the need and the treatment. Since these two 
factors have opposite effects, they cancel each other out and neither 
effect is visible without joint analysis of both variables. 

Measurement Approaches to Comhat Selection Bias 
When differential selection cannot be prevented, it must be identified 
by measuring the selection process and decisions. Differential impact 
analysis has the advantage of attacking from two different angles, using 
individual-level measures of preprogram characteristics and of the 
selection process experienced by each individual participant, and also 
employing agency-level implementation measures of selection policies 
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and typical practices. IndividuaMcvei mcasuitrs have several advantages 
for combating bias: 

1 . Ideally, they can include the agency *s diagnosis of each participant 
need and its service prescription for each participant as well as the 
treatment each individual actually received. Both the panicipant's 
true level of iH?ed and the agency's percej lion of each participant's 
jKred are important potential sources of selection bias. 

2. They offer information on the explicit selection process by the JTPA 
agency. For example, we can learn how much intake lime the agency 
spent with each particular participant. It is one thing to km)w what 
"fuir intake iiKiudcs Can agency-level measure) and another to 
know that individual A rectrived only a "Tast track" intake review, 
while individual B was judged to require extensive pre-employment 
serxices. 

3. They can include the route each specific individual takes into 
training. ITiis proves to be one primary indicator of selection bias. 

4. They allow precise measurement of program variants. incn?asing the 
jKiwer of the mcasunrs most im}K)rtant to any differential impact 
analysis. 

Agency-level variables also exhibit two particular strengths in com- 
bating selection bios. 

1. The probletn of compensatory ta'atment cannot be fully solved by 
individual-level measures, because no precise measures of need or 
assistance exi.>l. Agency-level measures indicate resources a\wl- 
ahlv or provided rw avera^ie. They are, therefore, much less influ- 
enced by compensatory treatment. For example, if agency A pro- 
vides job seaa'h assistance to only 5 percent of clients while agency 

2, B does so for 40 percent, it is almost certain that many individuals of 
equal need will receive this service in agency B but not in agency A. 
Agency policies directly affect selection. Targeting decisions, pol- 
icy toward "creaming/* policies regarding single vs, multiple activ- 
ity trcalments, and the like, have some consistent effect on selection 
across all participants enrolled through a particular agency. Such 
agency policies can indicate selection on difficult-to-iTK*asure crite- 
ria such as how participants present themselves inteipersonally. 
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The Potential far Seven Bias in 

Analyses of the Moa Basic Program Divisions 

Pcrhaj^i the two most basic program divisions manager might wish to 
anallyze using the differential impact analysis approi^h are different 
service provident within a service cteli very system and different pt>gram 
activity assignments. Unhappily, these divisions are the mc^t likely to be 
affected by selection bias an well as by bias from c(Mifoun<kd program 
variants. Implementation policies vary most widely acrc^s service pro- 
vi(ters and across basic program services. Service provider are likely to 
want control over which participants arc assigned to each basic service. 
Similarly* participants arc more likely to exercise choice regarding 
preferred basic services than about more specific implementation poli- 
cies. Service provider implementation and service mix involve basic 
resource allocation decisions and arc, 'hcrcfore, likely to be affected by 
geo{x>liticaI concerns. 

Different Serx ice Providers 

SDAs» and to a smaller extent their subcontractors, are located in 
different labor markets and political atmospheres. Although one can 
^ount for some of these diffcreiKes through measures of the labw 
market environment and agency policies, many will remain unmeasured. 
Therefore, some unknown degree of bias will persist in analysis across 
service providers, especially SDAs. The fuller atid more curate tte 
measures of labor market environment and agency policies concerning 
rrcruitment and selection, the smaller the remaining bias. 

Basic Program Activifies 

Basic service treatment options arc design^, in part, to accommodate 
differences in participant needs and qualifications* In jwticular, job 
search assistance assumes job readiness, OJT assumes minimum ^- 
ceptability to employers, and work exj^rience assumes an absence of 
even the most l»sic j<* experience. Selectio: ^s is likely to be 
especially sericHis in such cases, because diffv ' on the 

tesis of employability is explicitly called fot. '*erent 
treatments produce outcomes ihrmigh different iking 
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ihcm complex to compare directly. For example, true training intervcn- 
ticms are intended to pnxluce skill tran«;fer and employment, leading to 
a career line. These outcomes have little relevance, however, for an 
intervention based on .securing employment through the leverage of a 
payment made to employers. 

These concerns lead to the recommendation that {where resources 
allow) sample size should be large enough to accommodate separate 
analysis within each basic activity, along with tests performed across all 
activities. In addition, differential impact analysis should include 
^membersJiip-identifier*' variables indicating enrollment in each of the 
most common pmgram activities and in each SDA included in m 
analysis. These variables will absorb the effects of basic program activity 
and also some unmeasured selection efrect.s, thereby reducing bias in 
estimates of other effects. 

Although these problems appear overwhelming, and arc never sol zed 
completely, each additional measure of selection improves estimates. 
The analyst, therefore, has the power to produce highly useful and quite 
accurate estimates, which should, i^vertheless, always be interpreted as 
imperfect, Simpson's ( 1 989) analysis of one SDA illustrates the value of 
measuring selection into program activities. Without controls, it ap- 
peared that enrolling participants in multiple-sequenced activities pro- 
duced only a slight improvement in job retention at 1 3 weeks. However, 
after iiKluding in the analysis a set of competency bemrhmarks measured 
at program entry, a lai^e bem^ftt became observable for those enrolled in 
sequenmi activities. These competencies had been used as a basis for 
assigning multiple services, so that participants with greatest need 
received most intensive treatment. These two factors tended to cancel 
each other out, so that the effects of each could be obj^rved only when 
both were incluckd in the equation. 

Measuring Potential Influences on Outcomes 

DlfTerential impact analysis tests the impact of program variants and 
control variables on postprogram outcomes. In particular, managers may 
test whether fmrticular forms of implementation produce greater or lesser 
success and wtether assigntnent to particular program services or to 
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ser^iccH with particular qualities impmves an individuaPs chances of 
experiencing fK)sipn>grain success, Basic categories of factors may be 
tested to examine their possible impact on these outcomes. TTiey arc 
sdiown in exhibit 3 J. 

As discussed earlier, the analysis of how pivgram implementation and 
treatment influence outcomes is one of the most neglected areas of 
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Types Gf Factors to t» T^tsd &s Pussliils Ostsminanb 
of Pn^m ItotcDmM, wtOi Eampfsf 



Types trfOstMnlmmb 



Eiamptes 



Progfam 
varwnts: 
Forms 
program 



ooani^tjonal 
compon^ts 



• Form ot c^tractinB used 

• Program ciKt 

• Size of program 



impfementatton 



Service 
franwvon< 



• Tafgefiirg (seJeclion) polictes 

• Typical intaKe pmcedures 

• Quality controJ procedures 
» Exit pradices 



Progfam 
varmnts: 
Individua] 
services 



Membfirstup 
identifiers 



• Basic service assigned to (OJT. CT, etc.) 

• Senrice provider enrolteJ through 



(treatfmnt) 
received 



Vanable 
descriptions 



• Intake screening intensity or method 

• Treatment Intensity (tength. complexity) 

* Characten^tcsof thet^ner 

♦ Job search assistance received 



Qient 

characteristics 



♦ Age 

» Employment hrstory 

• Edicationai attainment 



Control 
varial>les 



Selection 
processes 



• tndfcators of "creaming* 

• Referrals involving prescreening 



labor market 
charactmtics 



• Unemt^oyment rates ^rvss place/time 

• Median ws^ iK^ws pteje/Bme 
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research in employment and training. Implementation studies do not 
tr^Iude estimates of impact on outcomes. Outcome studies typically 
vork with the very limited baj^ of trcaimcnt measures derived from 
accessible agency records. Tliis means the ihoughtful measurement of 
program variants is the key to creative advanc*es in program development 
based on differential impact analysis. 

Of the factors shown in exhibit 3.7, those labeled ''pa^gram varianis** 
are identified by choice, because the analyst hopes to learn whether 
particular implementation forms or particular individual services en- 
hance program outcomes. Program variants include tte basic organiza- 
tional arrangements and the service deliveiy framework established to 
implement delivery of services, and the specific ta'atment received by 
each participant. Treatment includes both variable liescripnons of the 
services nxeivcd, such as the length of a training pix)gram, and measures 
identifying only whether or not a participant was a member of some 
particular set. such as recipient of classroom training or enrollte through 
a particular service provider.** A third set of factors, "control variables/* 
is required in order to insure that effects estimated for program variants 
are as accurate as possible. These measure participant background, 
selection processes, labor market qualities, etc. When these are analyzed 
together, it becomes possible to isolate estimates of the unique elTects of 
each on a given pn>gi^m outcome. 

Tte conceptualization frfia^ of measuring treatment and implementa- 
tion is critical. The criteria summarized in cxhibh 3.8 offer some guid- 
amre. Criteria 1 and 6 are essentially technical, and may be honored 
without much knowledge of JTPA. However, criteria 2 through 5 require 
knowledge of the state or local JTPA .wrvice delivery system, making 
input from program managers critical to successful analysis. As a general 
principle, differential impact analysis heamies useful to ffuide program 
development only when measures are developed in collaboration he- 
n%Ten researchers and program directors. Measures are also suggested 
by JrPA implementation studies (reviewed by Grembowski. chapter 2 
in this volume) and some excellent analyses of CETA implementation 
(Levitan and Mangum 1981; Snedeker and Snedeker 1973; Franklin and 
Ripley 1984). 
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Criteria Ih^l Wtan SsMins tmplmaBntafliH) 
m ImSmtA tarMits tor A!tt^te 

1. Canthevarabtetmrn^ursdreltabiy? 

2. fs there reason to telievs it varms acro« mdividimls or service providers? 

3. Is there reason to beliew it represents a nontriwl program impaci? 

4. Do^ the variable m^isure a prt^ram \^riant under the control of program managers; that is. 
is it a policy-relevant variat>ie? 

5. Are program manners open to changing the program varanl to be n^sured? 

6. Given cost and time constraints, can the nmsure be integrated into a tteta collection «:herne? 

Service Provukr Surveys to Measure 
Program Implementation Variants 

In exhibit 3 J. the first ^ts of measures characterize program implem- 
entation — aspects of the organizations put in place to provide JTPA 
services. These measunss characterize the entire organization — its struc- 
ture, policies, and practices — rather than any one participant's treatment. 
Most program implementation variants are best measured through sur- 
veys of service-providing organizations. Data for each service provider 
can then be attached to the computer files of all participants who enrolled 
through that provider. In this way, data collected inexpensively by 
surveying a limited number of service providers can be used to analyze 
program imjxicts on all individual paiticii^ts in the sample. 

Depending on the nature of the measures, agency directors may be able 
to answer reliably, or agency staff may need to compare notes or consult 
records in order to characterize typical practices accurately. With easy- 
to-answcr questions, pf . surveys may be used. For more demaiuiing 
measures, however, a written survey, which allows time for data gather- 
ing, is preferable. The rreommendati(m here is to ase a written survey of 
each service-providing organization, with a backup telephone cc«itact 
person who can clarify questions as they arise. 

One problem with service provider surveys is that ageiKies may intend 
one form of implementation but actually carry out another, making j^lf- 
descriptions inaccurate. This can be partially remedied by a second form 
of implementation measurement: affgregated agency characteristics. 
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Aggregated variables are measure v^ith the individual panicif^int as unit 
and then summed^ percentaged^ or averaged aciros all participants within 
agency. For example* agency surveys could report whetter jwlicics 
emphasize training women for nontraditional (xrcupations. The aggre- 
gate form of n^suren^t for this same issue begins by constructing the 
iruiividual-level variable. Nontraditional training fields for women are 
identifie* and the training field of each female panicijMnt in the sample 
is codtd as traditional or ncHitraditicmal. That individual variable is then 
aggregated for each agency* producing an agency-level variable, *'per- 
cent of female partici]:mnts trained in nontraditional fields*" which may 
be used to double-check agency re}K)rts. 

Suggested Measures qf Frogmm impkmenkition 

Program implementation variants can be divided into bask organiza- 
tional components, such as fomis of contracting and staffing* and senive 
delivery framework within which intake* service assignment, provision 
of basic services and support services, and program exit occun TTie latter 
are most likely to have a direct influence on program outcomes, because 
ttey affect the nature of services provided and the selection prwess 
through which individuals are as^signed to treatments. Yet basic organ- 
izaticmal components are inexpensive to measure and may influence 
outcomes indirectly, by affi^ting various aspects of the ^rvice delivery 
ftBmework* Ttey are also important as control variables, lo protect 
interpretations from altcmaiive explanations after the research is 
cc»npleted* 

Bask Organizational Composition 

Forms of contracting. SDAs may or may not use requests for proposals 
(RFPs) as part of the subomtracting process. For both service fnovider 
contracts and trainer referral contracts* use of fix^-price* performance- 
ba^ contracts may be contrasted with otl^r appn^hes to ccmtnicting. 

Sti^ qualifications. Franklin and Ripley {,^>84) argue strongly that 
staff qualifications represent a key to success, although they not 
specific about what constitutes good qualifications. 

St(^ turnover. One might assume that staff stability (low turnover) 
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would predict success, although during iIk htc CETA era, one study 
fomid Uk? reverse to be true ^Simpson 1984a). 

Staff warklcHid and division of labor. Clicnt-to-slaff ratio, ongoing 
slafTtraining, and the division of staff between direct service, administni- 
lion, and development work can influence intensity and effectiveness of 
service delivery and, therefore, client outcomes. 

Service prtn ider history. The age of service providers, how much their 
services have changed over lime, their relations with the private and 
public sectors, and their rate of growth or decline may be useful to 
identify, although JTPA implementatitMi studies suggest these fetors 
have little differential impact on progi^ outcomes. 

Size. The size of SDA.i or subccMitractors (amount of grant, number of 
j^icipants, size of staff) may also be includoi as control variables. 

Program costs and cost-related policies. Program costs arc usually 
vcrydifficuhto measure precisely for imlividuals or for specific services. 
However, total cost-per-participant can be measun^ as an implementa- 
tion variable, and analyzed either as outcome or as one possible influence 
on outcomes, when different SDAs or different service providers are 
being compared. In addition, |»licies towatii use of support services, rate 
of employer reimbursement, and length of training can be measuiwi. 
Aggregated measures, such as the proportion of participants receiving 
support services or the average length of OJTs, may also be useful. 

Senice Delivery Framew ork 

Many of the same factors Grembowski poses as key to understanding 
organizational process (sec chapter 4) are also valuable measuies of 
program implementation for differential impact analysis. These may 
serve as program variants or as controls against bias. 

Explicit selection processes. Agency selection is critical to measure, 
both as a service quality issue and als^ because selection bias can be 
partially addressed with such measures. Agency policy may emphasize 
enrolling the most joh-i^dy, those with greatest need, or those whom the 
program is most likely to benefit. Agemry selection fx>licy may reserve 
some activities, such as short OJTs, primarily for those who are most 
easily served. A greater or smaller proportion of j^icipants may have 
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been referred initially by employers or sch(K)ls, adding un issue of 
preselection. 

Intake prm edurcs. Procedures used during intake for selection, diag- 
nosis, information giving, and counseling may difTer in intensity and 
type, n^y may aflfect how well the agency treatments match ihe abilities 
and needs of each participant to the labor maiicet. They may also act as 
indicators of the agency selecticm process, 

Pbssible measures include length and intensity of typical intake; 
proportion of participants who get full intake: number of "hurdles*" 
participants must pass, as an indicator of selection for motivation; 
whether intake is cenirali/ed or conductoi by subcontractors; whether 
mtake is conducted individually or in groups; what diagnostic tools are 
available and how often various tools arc used, policies regarding 
tai^eting and ott^r screening criteria; and what proportion of placements 
with employers were initiated by the employers. 

Quality control over referral and program activity mix. Service pro- 
viders may exercise strict control over the development of participant 
assignments, iiKluding rigorous screening of schools, agencies, or 
employers involved in treatment, or they may lake the laissez faire 
approach, offering information and encouraging participants' self-di- 
rected search for assignments, but exercising little control. This issue 
promises to be one of the most valuable areas for agency-level measure- 
ment, because referral represents the pivotal point of agency influence 
over treatment. 

Ptossible measures of agency control include whether employer refer- 
rals arc encouraged, giving contit)! to employers, or carefully reviewed 
and screened, retaining i^ency control; wl^ther policy encourages 
*'open contract'' referral arrangements, whereby employers agree to fill 
all openings for certain job titles by choosing among a set of eligibles sent 
for review by JTPA, giving service providers great control; whether tbe 
agency conducts formal quality reviews of employer-trainers or class- 
room-trailers, increasing control; and whetter a large pro]x>rti€m of 
assignments are developed through participants' self-directed search, 
which decreases agency ccmtroL 

The second half of the quality ccMitrol issue is how quality is defined. 
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AmcHig those agencies thai perform explicit quality control reviews over 
potential referrals or placements, what cnteria are used to define 
quality? Some passible measures include the following: previous JTPA 
pl^rement or retention track lecord, if a{^]icable; in tte case of em- 
ployer-based interventions^ employer stability or growth, typical non- 
JTPA turnover rates, typical wage rates, anK)unt and quality of training 
likely to occur in OJT assignments^ employer's ability to supervise con- 
stHKrtively; in the case of schools or community-based trainers, staff 
quality, ability to handle special student nwte, plm:ement assistance for 
graduates, and credibility among employers. 

Exit practices. TTw fiiuil element of trcatn^t is the set of program 
completion and job search c^icms implemented Exit practices are 
especially important to measure at the agencv level Individual measures 
of job search as istance i^ffer from tl^ ^^n^pensation problem: t}K)se 
least able to locate jobs on their own ^ mo.t likely to receive job search 
assistance, creating the appearance that p(^tprDgram employment is 
negatix'ely correlated with receipt of job search assistance. Agency-level 
measures of the average availability of ai:sistance are not aff^ted by the 
compensaticHi problem. Measures may include availability of job clubs 
or job search woricshops, jwportion of staff time devoted to postprogram 
job development, proportion of trainers who include formal ^b place- 
ment assistance, and method of placement, i.e., centralized or handled by 
subcontTKtors. 

Individml Measures of TreiOment Vanants 

The nature and intensity of services receivoJ by different fwticipants 
in a program vary widely within service providers as well as between 
them. These differences among mdividual experiences r^uire iiitdivid- 
ual level measurement. In addition, individual-level measures offer 
several ^vanta^s, as list^ in exhibit 3.9. 

Some individuals^ treatment experienc .s are routit^Iy reconted as i»urt 
of agerKry MIS files. Oti^rs may be recorded by agency staff as the 
treatments occur, or may be included in partici{^t follow-up surveys, 
measured through participants' r^all of the services they received. Hie 
preferable form of measurement is agency r^xwriing. Agency staff can 
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Ejdrtbiia.S 

• Tin y tte progmm services to mJtcon^ (or sam s{^iftc individuate, offering praise an2rfys*s 
0f the degree of association batvmn tfie two. 

• They tie sp«^ services and outcomes to stm:lf ic individual t>ackground cfiaraderisUcs* providing 
direct tests of control variables. 

• ftormally. they vaiy nwe wideJy than agency te\^ measures, strengthening statistitai tests. 

• Momnalfy. they suffer li^s overlap with other test vanabl^ than agenc>' eve) n^sur^, 
strengthening s^ticat tests. 



record services as they occur, avoiding recall errors, and including 
.^uemrc. Staff are also more able than participants to identify which 
service are administered. Agency measurement is also less expensive, 
as measured in terms of dollani, since it avoids telephone interview time. 
It is* however, more exj^nsive in terms of staff ame and often in terms 
of staff resistance to data recording. 

Gathering such information through participant surveys also has 
certain advantages: less lead time is n:quired: it is the only option 
available when samples arc identifsed at termination: and state or multi- 
SDA analyses may not reflect information necessary to coordinate data 
collection by large numbers of local direct service personnel m^dcing 
measurement at follow-up the only viable option. 

Intake Services, Screeninfi, and Selection 

Since intake intensity should be compen.«^tory. with greatest intensity 
reserved for those with greatest need, good intake will tend to equalize 
tl^ chances for success of those wub ^rt^ater and lesser need. Accurately 
estimating the impact of intake cn |?rogrnm outcomes, therefore* requires 
both measures of intake experiences md also measures of participants* 
need for intake assistance. The best approach to measuring ne^ is to 
i^tentify specific harriers to employment via the competency-bench- 
marking approach tkveloped most thoroughly in the area of youth 
competencies (Simpson 1989). 

Measures of individual intake experiences can include both the nature 
ami intensity of intake diagnosis and services. One approach wcmld 
measure time in intake (individually or in wc^kshops), specific intake 
diagnostics such as testing, and intensity of specific types of intake 
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services such as career counseling. Another approach would identify 
separate fhifhs taken hy inJMtiuals enroitte to program enroUment, such 
as einployer-iniiialed OJT extracts. 

Delay Betw een EHftihility and Enrolment 

The lime lag between eligibility and enniUmcnt may be a component 
of scleclitin bias. Enrollment that is almost immediate typically indicates 
a referral initiated by an employer or a school, and therefore involves 
preselection. Beyond that, up to some point, delays tend to weed out the 
least motivated. Very long delays, however, probably discourage those 
most qualified and motivated. 

Assignment to Basic Pm\>ram ActivUies 

Clearly, one measure of individual treatment must be the basic pro- 
gram activity or activitic to which each individual is assigned or 
referred. These are normally available through MIS files, although 
comparability of data elements may be an is.sue where multiple service 
providers are included in the analysis. Infonnation on multiple activities 
and sequencing will often rw^uirc an additional data collection effort 
beyond the standard MIS. It is especially important to distinguish 
between ( 1 ) midtiple-sequem ed activities planned in advance, such as 
an orientation workshop followed by classnxm n^ining. followed by 
OJT in the same skill area; and {2) "second chance" activities assigned 
to individuals who failed to utilize their first service successfully and are 
more likely to fail the .second time also. 

Treatment Intensity and Completion 

In addition to the type of program activity, the length and intensity of 
the activity should be measured, along with whether participanis< <w//j/f/e 
their programs. The most common measure here, length of time enrolled 
in JTPA, is easy to obtain fron MIS files, but confounds several 
incompatible measures: length of planned treatment, completion of 
treatment vs. dropout, addition of "second chance" treatment, trcaimem 
vs. dropout, addition of "second chanc*e'" treatment, and extension of 
enrollment in a postprogram administrative hold while employment is 
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sought. For sunic iif thcMr facli)rs. shorter cnn>llmcnt indicates pmbablc 
employment failure: tor others, longer enrollment indicates probable 
failure. They must, therefore, he measunrd sepanilely lo paxluce unam- 
biguous results. 

Unfortunately* no precise measures of (raininfi intensify per-time- 
period exist. However, partial indicators of intensity can be measured 
once the intended nature of the intervention is identified. These might 
include wliether the intervention results in a cr^^dentiaK in fields where 
one exists: how many hours of formal training and of "hands-on" training 
are provided: whether formal training is "mainstream" (satisfying slate 
certification, taken by nondisadvantaged individuals also): and if the 
intervention involves employment, how complex the job is and how 
much new material it presents for participants to learn. 

Ptrhaps the most important fomi of treatment intensity lo measure is 
the presence or absentee of multiple program activities assigned in 
sequence to address multiple barriers to employment. If pixxress analysis 
indicates that serving the hard-lo-?«rve is a priority for the organization, 
then this appreach is especially imjHiriant to analyze. 

Charucieri sties of Trainers 

Although factors such as traitiers* methods or organizational aironge- 
menis can seldom be changed by JTPA. knowknJge of which types of 
trainers most effectively produce desired outcomes can improve qi ality 
control and referral decisions. In addition, information on effective 
training appn>aches may be of interest to schools and especially to 
employers, since relatively little is know n about how to train effectively 
on the job. 

Some measures describing trainers can be gained frtmi participant 
follow-up surveys (Simpson 1982). However, the most reliable measure- 
ment sources are trainers themselves, i.e.. schools or employers. In 
classnx)m training, the easiest measures are typoU^^ies of trainers, e.g., 
trainers who enroll primarily JTPA participants vs. mainstream ti:ainers, 
or trainers who are public, proprietary, or community-based. In addition, 
trainers vary in size, mix between experiential ("hands-on**) and formal 
learning* inclusion of internships, and a great range of other characterise 
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tics. It is advisable to construct such measures in collabtiraiion with ItKal 
vocational educators, who an? mosi aware of variations in available 
training variantii. 

In the cases of employer-hased treatment, i .c., on-the-job training and. 
to a smaller extent, worii experience and youth tryout, a wide range of 
measures is available. Relatively little research has been tione. however, 
on such measures. Simpson's {1984a) Wa.shington Stale OJT research 
identified characicristics of the trainer and OJT pt>sitions, including 
employer's growth rate, typical non-JTPA turnover rate, the industrial 
sector, including whether public or private, the quality and complexity 
of the job. the use of relatively formal training methods, and rangt' of 
tranferahUiry of skills gained from training, i.e.. do they apply to a wide 
range of jobs or arc they "firm-specific"? 

E,xf>enditures per imiividual Partu ipani 

The primary marginal cost for each JTPA pJuticipant, is the dira t cost 
oftraininf*. Although other costs arc typically impossible to consider 
during a gross imjMct analysis, marginal training costs for each partici- 
pant arc usually available through contracts with trainers or employers. 
(See Zorwitsky c; al. 1985.) This means that the major program costs 
attached to e^ch specific particij^t could be analyzed if the.se records 
can be integrated with the basic data set being used. However, such 
analysis is not be confused with benefit-cost analysis. (See Johnson 
and Stromsdorfer, chapter 2 in this volume.) 

Another cost issue that also represents an agency policy issue is 
ancillary support services. Except for the issue of stipends offered during 
classroom training, there appears to be no cogent reason to detail specific 
support services. However, the total amount expended per person could 
be recorded at little cost. Such services might affect prt^ram completion, 
although no fx>stprogram impact would be expected beyond that caused 
by completion. 

Needs-based payments are complex to analyze. Income from any 
source has the potential to affect life stability, personal stress, and other 
factors, which can in turn influence postprogram labor maiisel experi- 
ences. Therefore, a precise analysis of the impact of needs-based sti- 
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pcnds per se requires measurcmcn! of lutul income during training as 
well as income from stipends. 

Program Exit and Job Search 

Agency implementation variables, discussed eariien measure the 
availability of various sup^ns sa tcrminaiion. Individual-level meas- 
ures indicate who makes use of which support services and provide the 
basis for aggregate agency-lcvcl variables. FH>ssible measures include a 
number of job «?arch services: enrollment in job club or less extensive 
job search courses, receipt of specific job referrals from trainers or JTPA 
staff, and receipt of less formal job search assistance fix>m JTPA staff. It 
is also useful to measure when the services occun^. If a job search 
worLshop occurred during or prior to training, all participants have bad 
benefit of it by the time they have to look for work. If the woilcshop occurs 
after the end of training, the most successful participants will not enroll 
because they will have found jobs already. This creates a "compensatory 
effect/* which is extrcmely difllcult to analyze validly. 

In addition, participants' Job search behaviors may be important to 
measure. These include the importance f^rticipanis place on finding or 
retaining work, using the skills learned during the JTPA treatment, and 
the extent to which a participant is ^'place bound*' (unable to relocate). 
The expressed importance of working in the training area can bt used as 
a control variable, but can also be analyzed as an intermediate }m>gram 
outcome. Those intending to use such analysis may wish to measure ll^ 
same variable at enmllmcnt in order to allow an estimate of change 
during training. 

Measurement of Contn^ Variables 

Asdiscussi^ earlier, the approach diflerentiaJ impact analyst*: takes to 
prevent bias involves measuring ^'control variables" for inclusion in 
multivariate statistical analysis. Many of the mc^t important protections 
against bias, such as measuring selection criteria, intake procedures, and 
exit practice*, are also of interest as imple5n;;nlation and treatment 
variants. Analysis of such measures serves the duai purposes of testing 
their impact on program outcomes and also testing whether their exclu- 
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sitHi from multivariate equations would bias estimated impact of otlwr 
prognini variants. Individual measures, such as preprogram barriers lo 
employment and the route Into the JTPA training activity, are also of 
Interest for proK.rammaiic reasons, but in dilTcrcntial impact analysis 
ihcy operate primarily as control variables. 

Other control variables lall into two categories: individual htick- 
gwund chararteristii's and the latH)r marker environmem . These meas- 
ures arc not analyzed in the hc^pe of impu>ving programs by changing 
them; most of them cannot be changed by program managers, or will not 
be changed, since they are part of the program mandate. TTiey are 
important because ihcy are likely to affect program outcomes and to 
differ across service pnwiders, program activities, or other program 
variants. Therefore, unless ihey arc measured and included in differen- 
tial impact equation.s. the estimated impact of program variants of 
interest to program managers can be biased. 

Sevenil individual background characteri.siics affect program out- 
comes: inheriteti characwristics. such as gender and ethnicity; previ- 
ously achieved t haracteristivs, such as education level and work expe- 
rience; and life cycle simadofi, such as marital status and number of 
dependents. Some mix of these measures is normally available in 
management information systems. Where factor; known to affect labor 
market experiences are omitted from MIS files, or where measurement 
is truncated to distinguish only program eligibles from noneligiblcs. 
MIS files must be augmented. Bonis (1979) provides a detailed enu- 
meration of individual background measures found to influence em- 
ployment status. Readere arc referred there for infonnation on the 
development of control variables measuring individual Inickground. 

C^e difficulty with preprogram measures is establishing their proper 
timeframe. "Preprogram dip" in earnings and empk>yment has been 
grappled with in much detail, making clear that information running 
back as far as three years befotie the program can be useful (Bloom and 
McLaughlin 1 982; Johnson and Stromsdorfer, chapter 2 in this volume ). 
That period may be too cmtly for agency data collection, but it is clear 
that too short a preprogram period, such an three or six months, will 
underestimate the long-term earning potential of many jmrticipants and 
will fail to distinguish those with temporary problems from others. 
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McaKuresof labor market characteristics are quite powerful in national 
studies. They appear to have less effect on train '"ng outcomes in a single 
state or locale. Even so. any study comparing service delivery in more 
than one get>graphic area must test the possibility that dilTerences in 
unemployment levels, average salary levels, or demand for particular 
types of jobs may affect estimates of differential impact analysis. Aside 
from census and employment data, the availability of labor market 
measures depends primarily on the role each state has played in devel- 
oping reliable occupational outlcwk data. Most SDAs have compiled this 
information during their planning jxriods. 

Construction of Category Identifiers 

The simplest but most important form of data any differential impact 
analysis requires is a set of ci^v numbers that identifies indmduuls and 
their membership in various cate^ones. These categories include sets of 
sicrvice providers, sjx^ific trainers, and types of services. Each category 
to be identified during analysis must have a unique identification 
number. TTiese are required — 

1 , To merge data from difl'ercnt sources, allowing the construction of 
data sets that include the full range of test and control variables, and 
allowing inexpensive service provider data to be inlegraled into in- 
dividual-level analysis. 

2, To organize the data set and know what original records to con.sult 
in ca.scs where errors on the computer file must be corrected. 

3, To CiMistruct membership identifier!*. These variables are the vehicle 
required before difTercniial impact analysis can test the impact of 
membership in particular organizational units, activities, or fields. 
(The construction of such variables is discussed below.) 

The identifiers listed below .should be included in any analysis which 
utilizes data from each source mentioned. The precise nature of each 
identifier depends on the common practice in the state or SDAs mounting 
the analysis effort. 

Participant identifiers. Codes identifying individual participants arc 
the basic data-file organizing unit and are also necessary in order to 
merge data from MIS files, follow-up interviews, and individual trcat- 
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menl rcconls. The best participant identifier is a social security number, 
which is unique and normally required if official data such as UI. welfare, 
or criminal justice information is to be combined with JTPA data. 

Employer identifiers. If employer interviews are conducted, data must 
be collected under an identifying code, unique for each employer, which 
is also recorded on each participant's file. In this way. the appropriate 
employer data may be added to the files of each individual. If agencies 
have not yet developed employer identification codes, they will also find 
them extremely useful for organizing employer relations and marketing 
information and assessing use patterns jmd retention track records of 
participating employers. 

Classroom traine,- identifiers. If special data are collected on class- 
room trainers, they must be catalogued un(ter identifiers also included <m 
participant records to allow data merging. In addition, trainers enrolling 
a sufficient number of participants in a sample may be tested using either/ 
or membership variables, if each trainer has a unique itentifier. 

Training field identifiers. The field in which partici}Kint5 trained or 
gained work experience should be identified. TTiis allows description of 
outcomes by field, construction of either/or membership variables where 
the number of cases allows, and introduction of labor market data tied to 
training field. 

SDA andsubi ontractor identifiers. When an analysis combines SDAs, 
each must be uniquely idenlifial in order to test for differences in 
outcomes produced by each and add labor market data to individual 
computer files. The same is true for subcontractor comparisons within 
one or more SDAs, If subcontractor* are numbered within each SDA. 
unique identifiers can be formed by combining SDA and subconn^ctor 
identifiers. 

Time peritni identifiers. TTic simplest reliable way to calculate time 
periods, such as lag between eligibility and enrollment, is to record the 
dale of each event, including eligibility, enrollment, treatment start, 
planned treatment end, actual treatment end, termination, and follow-up. 
If dates are expressed in compatible units, time pericxis can then be 
calculated by subtracting one from the otter. 
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Measures That Place Sj^kJDema^ an Sam/^g 

The measures of program variants just reviewed may be placed into 
fc^ categories, in onter to focus on two types that make special (temands 
on the size or structure of samples. These four apprraches are shown in 
exhibit 3. 10. The term ''variable description" refers to measures that may 
take a range of values, for example, percentages, amounts, oi dcgn^s of 
some quality. These arc the most common types of measun^s, explaining 
why measures arc often referred to as variables. TTicy are distinguished 
from '^membership identifiers/' which always take twily two values. The 
iiKlividual (1 ) does or (2) does not belong to that category. 

Standard sampling considerations arc structured for variables meas- 
ured at the individual level. The two-program implementation measures 
in exhibit 3. 1 0 plao; special demands on the sample. The demands differ 
depending on whether the measure is a membership identifier^ indicating 
whether or not a participant was served via some specific treatment 
ccMrtcxt, or a variable description of some particular aspect of program 
implementation. Membership identifiers, which indicate whether or not 
individuals received services via a particular organizational or treatment 
activity provide an effective way to locate impacts on postprogram 
outcomes, but not to explam why they are located where tl^y arc. 
Variable descriptions of prognmi characteristics, on the other hand, 
measure specific qualities that vary across all service providers or 
program activities, rather than separating each as a whole frwn the 
others. This approach docs not pinpoint concrete contexts where differ- 
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ences cKCur. However, it Iwlps explain why ihey cKcur, a quality which 
makes ihem especially helpful for program development. The jmrlicular 
demands placed cn sampling dilTer between the analysis of membership 
ideniifiers and the analysis of variable pmgram descriptions. 

Membership Identifiers 

Mcasun!s thai identify membership in particular !n:atment contexts 
place liule restriction on the number of service providers or other 
ireatmcni contexts required* If two contexts are identinable. they may be 
compared by entering the membership ideniificr into an equation that 
also includes the appn[)priaie control variables. However, this simplicity 
is gained at sc>me cost. Fii^t, explamtory (nm er is no jiireaur with 20 
conrexfx than with two. Each context is compared individually with all 
others. Second* reliahiliry of such amiparisons depends m the number 
of participants en roiled in each memirership araup. Thus, no matter how 
large the total sample, estimates for membership in a category containing 
very few participants cannot be rcliablc. This means that analysis of this 
type may require lai^e, disproportionately stmtified samples. 

Membership in higHy specific contexts, such as particular schools or 
employers, is usually immune to analysis because so few individuaJs 
belong to each context. Membership in larger units, such as SDAs, is 
entirely possible to assess given thai a large enough sample is drawn from 
each SDA. TTiis limitation on the analysis of specific treatment contexts 
is of greatest interest for states and large SDAs. which may wish to as.scss 
relative performance among differxjnt units within the system. 

Variable Descriptions of Profiram Implementation 
The sampling cfcmands made by variable descriptions of implementa- 
tion are directly opposite to those made by membership identifiers; they 
requite multiple treatment contexts, but not larger samples within each. 
Since participants in each context receive a specific value on some 
measurement scale, the number of participants in each context mmters 
little. Thus, sample stratification is unnecessary. TTiat advantage, how- 
ever, is purchased at the cost of requiring multiple treatment contexts to 
be included in the sample. 
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Imagine that two scrviCC providers have been mcasurvd on two 
variabieK — intensity of intake pnx*edures and the degree of job «?aa*h 
asHistance provide. If wc find that the two providers differ in outcome 
kvcL how can we decide which of these variables accounts for the 
difference? For that matter, how can we claim that citlHrr of these, rather 
than some other variable, explains the difference? To assess wheiter 
intake or placement had the imjmcl. we need lo compare situations 
characterized by thorough intake but little job search assistance, and vice 
versa. With only two organizations, however, that is not possible. TTksc 
two agency characteristics, as well as any others one can imagine, arc by 
definition perfectly correlated and cannot be disentangled. (In statistical 
terms, only one degree of frecdcMii is available.) 

This same pn>blem faces it:?search cximparing more than two contexts, 
wbciTc the variable in question hap]:^ns to differentiate only one from all 
others. An analysis reported by Franklin and Ripley (1984) illustrates. 
Ttey report that program performance: was lower in CETA prime 
sponsors characterized by "crisis management** style. While this finding 
appears reasonable, only one prime sjwn.sor in their sample was so 
characterized. Their conclusion, ihei^fore, was based on a compari.son 
between one prime sponsor and 14 others. Consequently, any number of 
other qualities of thai one prime sponsor could have pnxluccd the 
differences tlwy observed. 

In the ca.se where three service providers are included in a s^unplc, it 
is very likely that the problems discussed above will remain. However, 
there is no*-v a pi>ssibility that, in unusual circumstances, one variable 
characteristic of service providers would have such a stipng and consis- 
tent impact that a statistically reliable effect would emerge. The principle 
of fwsimony — using the simplest explanation consistent with the facts — 
becomes the guide to inteipretalion here. If ti^ ditTerences among 
outcomes in the three contexts cloi^ly fit a single linear treatment 
measure* but no others, then it is parsimonious to explain findings with 
that one factor. If, however, they vary far from a linear fit, or if more than 
one variable fits equally well, the less tidy, but more acx*urate interpn^- 
tation must be used; namely, that each unit differs from the other for 
reasons we cannot demonstrate. 
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If we iniroduccd a second tiraUncnt characteristic vixiable into the 
analysis based on three contexts, we would automaticaily rcvcrt lo the 
case in which it is impossible to distinguish among competing explana- 
tions. In statistical terms, the number of variables that may l>e uniqu«:Iy 
estimated may not be greater tiian the degrees of frecitom, which equal 
the number of cases minus one. Since these variables arc measured only 
ai the organizational leveK the number of cases we are speaking about is 
the number of service-providing organizations in the an'ilysis. 

Extending this line of thought, it is apparent that analysis including 
many SDAs or analysis of large SDAs including many service providers 
can be especially valuable for local pnogram tfcvelopment. Tl^ Im^r the 
number of different agcmries in the analysis, the more feasible tte tests 
of agency implementation vari ables. More variables can be handled 
simultaneously, and each is tested more reliably and less ambiguously. 
TTiat is, other things equal the more separate service providers included 
in a sample, the lower the covariancc among implementation variants is 
likely to be, strengthening the ability of multivariate analysis to estimate 
the unique effects of each. 

TTiis general mie, that the larger the number of contexts, the firmer the 
analysis of variable program characteristics, Icmis to a practical question: 
what is the minimum number of service prtwiden* required for a reason- 
able differential impaci analysis of agency-level implementation meas- 
ures? TTie answer is twofold. First, the bad t^ws. TTie answer depends 
on many factors: variance in each independent variable, variance in the 
outcome variable, covariance among independent variables, and covari- 
ance between independent variables and the outcome variable. There- 
fore, no piwise minimum can be set forth. One might reasonably say that 
there is little point in pursuing analysis of variable program implemen- 
tation measures with fewer than six or s^ven service providers. In many 
cases this would be too few, while in others, it would be sufficient. 

Second, the good news. There is an analysis procedure that can in most 
cases protect against incorrectly attributing too mucb importance to 
variable descriptions of program characterif:lics. Tnis procedure in- 
volves jointly testing ih variable program characteristic measures along 
with membership identifiers indicating enrollment in each particular 
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service provider in ihe sample. After identifying variable program 
characteriHtici; that apfxrar Ktati.stically rcliablc, the analyst then adds to 
the equation the set of membership identifiers indicating enrollment 
through each specific service provider 

If the variable program characteristics retain their statistically reliable 
effects, then our ccmfidence in the initial findings itrmains high. If tteir 
effect in the cquaiion is eliminated by the addition of the membership 
identifiers, then we must coiKlude either that some service providers 
differ from each other, but we do not know why, or that the initial test 
procedure was inappropriate* With small numbers of units, the latter is 
always a strong likelihtxxi. 

Analysis Procedures for I]^riptive 
Outcome and DifTerentiai Impact Evaluations 

This chajMer makes no attempt to prnvide instmction in the use of 
statistics. However, a brief overview of analysis strategies for descriptive 
gross outcomes analysis and ditfercntial imj^t analysis may be useful. 

Desiripfivc analysis involves quite basic statistical tools. TImj value of 
descriptive analysis rests more on the thought that goes into the questions 
the analyst asks than on statistical .vophisticaticm. Descriptive analysis 
begins with univariate (one-v^ablc) averages or percentage distribu- 
tiCTis. BeycMKi that, bivariate (iwo-variable) associations can be calcu- 
lated, as long as the analyst bears in mind that ctescrijMive associations 
can be produced by many factors other than the two being analyzed. 
Exhibit 3. 1 1 describes conditions under which different bivariate statis- 
tics are most appropriate. 

Differential im}^t analysis can be performed satisfactorily with 
standaiti multiple regression techniques, except for one particul^ situ- 
aticm, which is discussed. The strategy of multivariate analysis is 
straightforward. One outcome if analyzed, \ . ith multiple potential influ- 
ences tested simultaneously to estimate the unique imj^t of each on the 
outcome. In this instance, tl^ gc^l is to af wrtain whetl^r ami how much 
policy variables of interest affect the outcome after taking into aconmt 
the possible effects of other factors such as selection. However, the 
statistical techniques required to implen^nt that strategy require special- 
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izcd iratning. Pwns of ihc discussion thai follows assume prior back- 
ground iti multivariate analysts. 

Analyzing Various Types o/Mmsares 

A wide range of statistics is available in various software packages. 
However, i^arly all statistical tests required for dcMniptive gross impact 
analy:%is or dincrential impact analysis can be performed with four basic 
tools: chi square, analysis of variance ( ANOVAK Pearson conelalion. 
and ordinary least squares (OLS) regression. Why tt^se are typically 
adequate is laid out in a highly readable form in Bjomsiadt and Knoke 
( 1982). Which of these is used depends on the nature of the analysis and 
the way in which the variable was measured — commonly referred to as 
the level of measun^ment. Exhibit 3. 1 1 suggests appropriate statistics for 
different levels of meastuement and for different analysis goals. 

The critical distinction regarding level of measurement is between 
<>rdered and nominal variables. Ordered vimablcs arc those for which 
values assigned to each category of the variable form a logically 
defensible s<^uence from smaller to larger, lower to higher, etc. Ordered 
variables include age, level of satisfaction^ costs, raiingH on various 
descriptive scales, and the like. 

Variables that cannot be ordered arc termed nominal variables. The 
categories of variables like marital status or SDA identiftcation codes 
cannot be placed in a meaningful hierarchy or sequence. The results of 
tests that require ordered variables would be meaningless if applied to 
nominal variables such as th^^se. 

Dichotomous variables, those taking only two values, such as "yes'* 
and **no/' occupy a special status in that they are by definition ordered, 
even when they appear logically nonorderable. Any variable that in- 
cludes only two values can be expressed as a yes/no question. In the case 
of one SDA vs. others, for example, the variable becomes "Did this 
{^icipant enroll through SDA #1 ?** The respwises '*yes** and *W are 
inierprelable as ordered, with yes greater than no. It is this quality of 
dichotomies that makes membership identifiei^ especially powerful in 
differential imj^t analysis. 

Statist seal assumptions vary somewhat for dependent (outcotne) vari- 
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ExiiMI3.11 
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ables vs. independent (pwlicior, explanatory) variables. Therefore, tte 
choice of statistical fools depends cmi the level of measurement for each. 
Exhibit 3. 1 1 reflects this requirement. Analysts goals are separata} into 
bivariate (two* variable) and multivariate (one dependent variable, more 
than one ir^pendent variable) cases, with measurement indicated for 
both independent and dependent variables. 




194 Evaluating Social Probiemii 

MmUivttn^ Amly^ wMt m l^ktOomous De^ndem VarialHe 

Evcfy statistic is (kvelopcd on the basis of mathematical assumptions. 
In the case of ordinary least suuarcs rcgn^si^m, many of the original 
restrictive assumptions have proven unne^ssi^y. That is. the statistic is 
highly robust; data can be structured in ways not fully satisfying 
statistical assumptions, yet the statistic produces accurate and efficient 
estimates. Even so, in tte case wtere ih: dependent variable is dichoto- 
mous an j is highly skeweCi (unevenly distributed), assumptions are vio- 
lau i 'leverciy am] enx>r cm rtmh. 

Happily, rei ent work with statistics based on log-linear transforma- 
tions of dichm^MncHis depemient variables — am) using '"maximum like*- 
lihood ciii sqmrc goodi^ss of fit*^ tests — avoid iIk problems faced by 
ordinary regression. This means that appropriate conservative multivari- 
ate method? to analyze dichotomies, such as whether or not participants 
arc employed at follow-up, do exist. RegrcssitHi tests have been com- 
pared with these mom c(m»ervative methods, with tt^ result that we can 
now be quite certain when we are required to use the more conservative, 
but also less convenient, methods and when the simpler regression 
analysts is appropriate. (See Knoke 1975: Goodman 1976; Giilespi 
1977.) The following guidelines summarize this knowiedge: 

1. !f a dichotomous riependcnt variable is split relatively evenly (be- 
tween 75 pcrceiif/25 percent ami 25 perccni/75 percent) OLS regres- 
sion may be used. 

2. If OLS regression can*^ot be used and aU or many independent 
variables are orders, Logit or Probit transfcmnations of the depeiKl- 
ent variable are advisable. 

3. If OLS regression cannot be used and many independent variables 
are dicbotonwui or nominal, Goodman's Muhiway Cmlingency 
Table Analysis may be used, 

Cmstmcting and Testing 

Membership t4kntifkrs C'Dummy Variables''} 

Some extremely valuable factors to include as independent variables 
in multiple regression analysis aic measured as nominal variables, which 
arc not admissible in a regression equation. Nominal variables may be 
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analyzed, however, after Ihcy are tnuiKfonned into dichotomies that are 
tested in place of the original variable. These dichotomies, known as 
"dummy variables" in formal statistical analysis, are called nwimbcrship 
identifiers throughout this chapter to imiicate Jicir nature, i.e., they 
measure whether or not a participant belongs to a f^rticular category. For 
example. MIS system.s may include a measure of ethnicity, including 
values for each of five or more major groups. If membership variables arc 
constructed for each of these groups (e.g., a variable named "olhwhitc," 
fuid scored I for all origini^ly coded "other white" and 0 for all others), 
these new dichotomies may be tested as independent variables in 
regressi(Mi analysis. 

Regression slopes thai result from tests of dummy variables, if 
statistically reliable, indicate thai mcmbcn; of the named group (e.g., 
those enrolled through unit ^4, "other white** parti ;ipants. or OJT partici- 
pants) are the estimated average amount higher cr lower on the outcome 
in question than all members of other groups. All but one of the dummy 
variables created from an original nominal variable may be tested in a 
single equation. 

Constructing and Testing Interaction Terms 

One useful type of question for JTPA pn>gram managers may be 
a(^res3«i by using interaction terms. Tliis qt^stion is Do particular 
groups of {Mrticipants. more than others, experience greater success from 
some program varianjs than from other variants? For example, is 
classroom training (CT) norc successful than other treatments in era.sing 
the deflcit product by previous low educational attainment? In an 
interaction, two variables amihine to produce a joint effect diffenmt 
from that which both m;ting independently would produce. In the 
example, dn^ping out of high school reduces poslprcgram outcomes 
and CT may, in itself, produce higher or lower than average outcomes. 
In addition, the interaction hypothesis suggests th^ the imp^t of 
eitucational attainment m postprogram outcomes is stronger when tix 
treatment is not classroom training than when it is. To test such an 
hypothesis. oi« must con.struct an imer^tion term that would be scored 
I for dropouts who enroll in CT and 0 for all others. This interactiOT term. 
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4hc product of the other two, identifies those ^ *dividuals who were iii a 
position to have some fxmion of their educational deficit eliminated by 
participation in CT, 

Normally, the interai^tion term* appropriate control vi riablta, ami tte 
original v^nables from which the inter^tion was constructed must all be 
included in one multiple regression equation (Blalock 1965), Since the 
interaction term will include portions of both original variables^ an effect 
of cither or both original variables would em>neously be carried by the 
interaction term alone. Only when the original terms are both included 
in the test can we be certain an effect of the interaction term is not 
spurious.^* If such a test produced, for example, a slope of -0.20 for 
dropout status and -fO.lO for the interaction term, then the proper 
interpretation would be (a) that being a high schiwl dropout, in itself, 
reduces posiprogram employment 20 percent, and (b) that Cf erases half 
of the education effect among dropouts, so that drofwuts who enrolled in 
CT experience only a 10 percent lower placement rate (-0.20 -K)JO) 

Re^rting Standardized or UnstandanUzed Regression Caefftcients 
Standardized regression coefTicients, termed **Belas,'* are often re- 
ported because they indicate the relative power of each variable in one 
equation to account for variation in the dependent variable. Betas have 
a commonscnsc meaning similar to that of a correlatiim: a Beta of 0.5 
always indicates a "stronger" effect \\m\ a Beta of 0.4. UnstandanJizcd 
coefficients (regression slopes) arc expressed in terms of the metric of the 
inde}K:ndent and dependent variables, and arc much more precise but 
often less intuitively satisfying. If, for example, education is scored using 
a four-point .scale, a slope of 0.10 inJicales that each step of that 
education scale raises the outcome variable by OJO. The lowest step 
compared with the highest, thr^ in!erval.s above, has an estimated 0 JO 
higter level If the dependent variable is employment status, 0.30 
translates to 30 percent. If it is hourly wage, 0.30 translates to 30 cents. 

Tliese considerations make sloj^s stmicwhat more complex than 
Betas to communicate when Findings arc reported. When results of 
research are being applied to progmm development efforts, however, 
unslandardized slopes are the preferable estimate because they give a 
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direct estimate of the ammmt of change in an outamie rfuit is pnHJm'ed 
by a given change in the input. Is it more helpful for me to know thai 
education* which 1 cannot ccHitrol is more powerful than program 
^tiviiy? Or am I better scrvai by estimating that i**'*' the elTects of 
education are s^rcounted fOi\, my QJT program produces postprograni 
wages an estimated 47 cents lower (or higher) than my CT program? 
Clearly the latter, unstandardim] report is preferable. 

In addition, unlike standardi2ed Betas, unstandardized slopes are not 
influenced by the variance of a particular variable within a sample. This 
can be important when relatively small subgroups are being analyzed. If 
only 10 percent of my sample enrolled in work experience, even if WEX 
participants are employed only half as often as otters, the variable 
-enrolled in WEX?*' can explain only a limited portion of the variation 
in employment experienoxi by the entire s;unplc. However, the unstan- 
dardized slope indicates how much less often this minority is employed 
than arc other [^ici}nuits. The slope remains the same whether 10 
percent or 50 percent of participants arc enrolled in work experience. 

In general, unstandanitzed regression slopes are both more useful and 
easier to report when ( 1 ) the dependent variable is naturally inlcrpreiablc, 
as in the case of income or a dichotomy that translates to percentages, and 
(2) the independent variable is a dichotomy, allowing statements like 
**partjcii^ts in category /4 are X percent higher than those in category S/* 
In other cases, tl^ analyst must choose between reporting ease aiul 
managerial usefulness. For a fjU analysis, both forms augment each 
other. For example, a report might indicate thai a particular program 
service has a negligible impact on variation in outcomes for an entire 
SDA. but go on to show a large impact on a few clients. 

Estimating Change 

For descriptive analysis, change may be indicated by subtracting tlie 
prepivgram value from the {wstprogram value of an identically measured 
variable. This procedure is simple and the results are often taken at face 
valw. However, the descriptive report of change is especially problem- 
atic because change is heavily def^ndent on the original base figure. If, 
for example, a sample includes mar stt dents or displaced homemaken; 
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with zero earnings during the prepr^;rani year, then regardless of 
program impact, that group is likely to generate higter change in 
earnings than a group including primarily high previous ^ners. At the 
other end of the scale, displai^-worker pn^rams will typically sIk)w 
negative change Hguies — ^a reduction in earnings — noi because ttey are 
less effective than other progi^ams. but because preprogram earnings 
were especially high. This proWera is especially severe if prepn^ram 
values vary more widely than postpDgram values, as is more often die 
case in highly successful programs than in less successful programs. 
For this reason, presentation of descriptive change results can be seri- 
(Hisly misleading. 

In multivariate differential impact analysis, the goal is to estimate 
unique causal effects of each fac^':^r lesi^. Since the prepn^nun level of 
an outcome variable clearly affects change in that variable, any analysis 
of change must include that level as a predictor in the regression 
equation. This necessity l^ds to an approach that considerably improves 
the ease and usefulness of ct^ge analysis. Predicting change with 
preprogram level prcxiuces awkward results, i.e„ higher preprogram' 
levels of any variable produce lower rates of change. Hov;ever, it is 
possible (and preferable) to replace change with postprogram level as the 
outcome being predicted. Sin*^ change is calculated as postprogram 
minus preprogram level, any two of the following variables is suHicient 
to produce the third via simple mathematics operations: (1) the 
preprogram level, (2) the postprogram level, and (3) change. Therefore, 
the equation is satisfied regardless of which two are used. 

It is preferable to use the two that produce the most sensible results: the 
postprc^ram outcome as dependent variable and the preprogram meas- 
ure of the same outcome as a control variable. In such a case, tte effect 
of preprogram on postprogram level (the autoregression term) indicates 
stability over time, i.e., the temtency for those most employable before 
the program to be most employable after the program also. Other 
variables in the equation that show a reliable imf^ct on t^ postprogram 
outcome may be correctly interpreted as indicating factors that increase 
or decrease (change) the outcome in question from the preprogram to the 
postprogram pcricxl. 
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Hiese considerations open am>ther possibility. It is a short step to 
ccHK^iucte thai Lhe multiv^ate analysis of change is hurt very little if the 
prqm)gram measure differs slightly from the {wallel postprogram 
n^asuie. The analysis still indices which factors irnnease or decrease 
ti» outcome controlling for prejHogram level of approximately the same 
fM^tor. Ickntical pre- ami postprogram measures are desirable. However, 
the analysis of change remains possible with less-than-exactly parallel 
pre- md postprogram measures. 

Amlyzing a Lm^ Number ofPi^ndal l¥^fluenc€s m Outcomes 

Multivariate analysis is straightforward in stuuies wt^re only a few 
tl^retically derived variables are tested. All are entered into the equa- 
tion and the results reported. In cases wtere many measures arc to be 
tested as independent variables, however, it is no longer possible or 
disable to include all in a single test. Such anempts can make undue 
demands on the sample size, a problem that becomes especially serious 
if the i^mple size is reduced by the accumulation of missing cases from 
each of tl^ many variables involved. Hits produce the problem of how 
to move through multiple tests efficiently withtnit distorting or overlook- 
ing effects. Tte following suggestions may assist in that encteavor. 
Performing such analysis^ of cou!^« requires prior statistical back- 
ground* This "cocrfcbook^ summary is not meant to imply otherwise. It 
only sugges>':!> steps to make analysis relatively efficient. 

Step I. Insure that vari^les are in the proper form for multivariate 
analysis and that variation is sufficient to make analysis meaningfuL For 
data management jnirpmes it is advisable to construct a "^coctebook^" 
li^g all variables and showii^ for each the level of n^asurement^ 
number of useable c^ses. aiKi an indicator of variation. 

Step 2. Select the appropriate dependent variable for each analysis. 

Step 3. Separate variables according to their i^ipoiiance to the analy- 
sis* Those that are most important, brcause they ,ire known to attm tte 
outcome and must be included to prevent bias or because they hold 
special progran^ developm^t interest, shculd b^ given priority during 
analysis. 

Step 4. Separate variables according to missing cases. In particular^ 
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questiom asked only of sub^ts of parttci^^th, such as job qualities or 
reaKons for unemployment, KhtKild be analyzed se|>arately from other 
questions applying to all. Hie safest form of multivariate analysis is 
based only on cases for which full infomiation on all variables is present. 
Under that approach, any case with a single missing value is eliminated 
from the entin; analysis. 

Step 5. Compute correlations between all independent variables and 
the dependent variable being analyzed. Conrelalions are the tesis for the 
calculation of multiple regression coefficients, making them the appro- 
priate bivariate test building toward rcgression analysis. 

Step 6, Identify those variables that are appropriate in terms of missing 
values and have high priority as control variables — those i\H]uired to 
protect against biased estimates. Ot^rve their correlations with the 
outcome(s) in question. Select from this set those variables exhibiting a 
reliable association with the outcome being analyzed. 

Sfep 7. Enter the variables selected at the conclusion of step 6 into a 
multiple regression equation, to identify the subset of these variables that 
have reliable multivariate effects on the outcome. For simplicity, ana- 
lysts may use a stepwise procedure, which automatically selects leliable 
effects. This produces a minimal scl of control variables which must be 
included in subsequent runs. Other variables from tl^se tests may be set 
aside for Ibc moment with tl^ knowledge that were they included in the 
regression equation, their effect would be too small to alter findings 
noticeably. 

Step Identify the most important test variables, i.e., program 
variants of special policy interest. Observe their correlations with the 
outcome in question, selecting th(n»e showing reliable association, l^ter 
these singly, or in appropriate sets, into equaticMis that iiKlude the 
minimal set of control variables identified in step 7. 

Step 9. In addition to variables tested in step 8, analysts may wish to 
explore other program variants, hoping to discover useful unexpected 
relationships. Group measures according to policy area, such as intake, 
quality control, or trainer characteristics. Correlate these with the out- 
come being analyzed and enter thoj^ which are reliably greater than zero 
into an equation including the minimal set of control variables identified 
in step 7. 
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Step 10, The jroccdures outlii^d above for reducing the set of reliable 
effects ignores the possibility of sHppresshm. a situation in which two 
variables are correlated with each other but have opfK)site effects on the 
dependent variables and, therefore:, tend to cancel each otl^r out in 
bivariate tests. TTicse effects become visible only when both indepeiMtent 
variables are tested jointly. They are, therefore, overlooked when only 
reliable bivariate correlations are forw^ed for test in regression cqua- 
tions, as in steps 6-9. 

Short of a full exposition of this issue, one step may be suggested to 
guard against most errors of this type. Suppressitm of the type tl^ analyst 
most wishes to uncover occurs only when some variable is ccmielated 
with one of the variables iiteniifiad during steps 8 and 9 as r Mq 
predictors. Correlations should, tterefore, be calculated between each of 
these reliable effects and other independent variables. Wheie reliable 
correlations arc found, the variable in question may be added to the 
reduced set of reliable effects Iwated after step 8 or 9, Relatively few 
changes will be product by such a procedure, but it does guard against 
the most damaging errors from undetected relationships* These tests may 
be facilitated by using a backwards stepwise elimination of unreliable 
effects^ where statistical packages include this option. 

Step I L Membership identifier variables, such as service provider, 
indusirial s n:tor, and similar ethers, should be examined if they have ncrt 
a^neady been included as program vmiants. H^se may be ^cted to the 
rt luced sets of reliable effects identified in steps 8, 9, or 10. Findings 
may pii>''e useful for future contracting or marketing. Also, such tests 
protect against spurious findings of program effects. 

Step 12, Finally, having identified reduced sets of the most powerful 
and unique effects on e^h outc«*me being analyzed, the aralysl will be 
well-advised to letum to the data set in order to examine what measures 
are associated with the,%e key effects. Such analyses may be conducted 
formally, using the% key effects as dependent variables in their own 
right, or may be undertaken as less formalized examinations of patterned 
associations. Such furlt^r analysis can corroborate or challenge inhial 
interpretations, or can help the analyst develop interpretations of initial 
findings by detailing the ap]^rent nature of the variables found to have 
greatest impact on ihc outcome. 
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Cabining Ikitafrmn I^ereru Sources 

complexity of effective differential imp^t analysis is the need to 
combine data from several different sourras into a tailor-m^e data set. 
Tte combination of data also provides two of tl^ method's stiengths: 
ll^ ability to protect against ^lecticHi bias from several angles, and tte 
ability to measure different types of variables in the most reliable and 
efficient manner. 

Aside from the availability of appropriate computer f acilities« the key 
to conioining (merging) data is to include the airrect icteniifiers in each 
data set to be mei^ged. It is advisable to produce a nuister identifier cover 
sheet to become of each {^iticipant dua file. Hiis si^t ^^hould 
include all the identifying informatitMi required to merge data: partici- 
pant social security number, SDA ittentiHer* agency (subccmtnu:tor) 
identifier, employer identifier, etc. 

For differential impact analysis, all data shois d he merged into 
individual fxirticipam records^ siiu" the participant is the unit of analy- 
sis. All identifiers must appear in the panicipant's original data file. Each 
of the other original files must contain only t!^ particular identifier 
required ij correctly merge into the participant file. For example, 
implementation program variants are measured at tte agency level Each 
participant who enrolled through the agency with the ID code ** 1 0" will 
receive values on all imptementatim variaMes which wm provicfed by 
that agemry. Tte agency identifier ^vill a|^ar on those jmrticipants* 
master identifier sl^ts and also m the appropriate agency implemen- 
tation data reports, allowing the match of identifiers, followed by the 
combination of data. 

Once data sets are merged, statistical tests will be calculated m the 
basis of the number of participants (or employers) in Uie data set, not on 
the basis of the number of service providers or geographical regions that 
may have supplied fmrticulardata elements. Tk? analyst must, therefore^ 
remain awaie of limitations surrounding the number of separate treat- 
ment contexts required for reliable differential impact tests (discussed 
earlier). 
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Research De^gn for an Employer FoUow-Up 

With advent of JTPA and ibe exp^msion of private sector involve- 
ment, interest in measuring employer benefits has risen. The popularity 
of this issue among service provicters is no doubt connected to a concern 
for mailceting JTPA services and products to employers. Tlw perceptions 
of OJT or WEX employers arc useful to iiklicate which progi^ 
approaches arc relatively effective, which are distasteful to employers, 
and what steps might encourage or discourage future participation by 
employers (e.g., Wentling and Lawson 1975; Minnesota Office of 
Statewide CETA Coordination 1979; Simpson 1984b >. Employers may 
be interviewed primarily as a maiiteting tool. In addition, employer 
interviews can be valuable progi^ evaluaticm tools. They may be used 
to assess tte effeciivei^s of partici{^t services or to asse^ employer 
benefits, both central to JTPA prc^ram development. How thest goals 
translate into a research design depends on the relation of the employers 
being studied to JTPA, in particular, whether they are participating 
employers or termination employers. 

Termination employers are those who employed participants at iteir 
terminatimi from any program activity. They are coasumers of JTPA's 
"products." Some hire former JTPA participants without knowing that 
the training their new employees received was supported by JTPA. 
(Xters have {wticipated in providing training or experience tu the 
]»rticipants they subsequently hire at termination. 

Participating employers are those who {^icipated in the delivery of 
services, thnsugh on-the-job training, woilc experience, or tryoul, regard- 
less of the termination status of the frarticipanis involved. Many become 
tennination employers also. However, many employers j^rticipate in 
contracts that end prematurely, or complete a contract but choose mH to 
hire the JTPA participant following his or her participation in JTPA. 

For termination employers, surveys may ask direct marketing ques- 
tions, such as how the employer came to hire a JTPA product, or may 
address indirect marketinn goais, such as measuring employer satisfac- 
tion with the former participant hired, with tte goal of demonstrating 
program success to future employers. For participating employers, 
maiiceting questions may be expaiMled to include willingness to continue 
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or expand future participalion. In this case, the goals of marketing the 
program and assessing employer costs or benefits overlap. The major 
marketing tool for engaging new employers in «?rvicc delivery is evi- 
dence that past participating employers feel that the benefits of partici- 
pating have outweighed the costs. 

For termination employers, the goals of assessing participant services 
and employer benefits are nearly identical. Employer costs or benefits 
occur entirely because participants are or arc not well-prepared for tl^ir 
jobs. In the c4lsc of panicipaling employers, reports of costs and benefits 
can include not only ratings of the panicipant. but also evaluations of the 
JTPA programs and personnel and direct perceptions of participation as 
beneficial or costly. 

Employer surveys are unique in that many employer re}x»rls may be 
taken at fm:e value and arc. therefore. espt*cially useful for descriptive 
analysis. The employer's role as consumer of JTPA pixxlucts (partici- 
pants), makes employer ratings of participants valuable to JTPA pro- 
gram operators regardless of the factors infiuencing them. Because JTPA 
agencies wish to ha /c the most effective participating employers repeat 
their involvement, employer saiisfaciitw with agency policies or person- 
nel is critical. Similarly, employers* pcreepiions that ihey have berefited 
from participating in delivering JTPA services are meaningful descrip- 
tive estimates of employer costs and benefits. 

In addition, information gathered from employers can be valuable to 
the analysis of panicij^t outcomes. Employers are in a unique jxjsition 
to report subtle job quality outcomes, certain forms of selection, and for 
participating employers, individual-level measures of training provided 
or other qualities of the program intervention. 

Identifying Employ er Population to 
Be Analyzed and Desifrning the Sample 

T?ie most tesic of all employer design questions is whether the 
population being studied includes all temiinaticm employers, all partici- 
pating employers, or both. In widition, managers who wish an in-depth 
analysis of one .specific program activity may prefer an even more 
specific definition. .>uwh as all OJT employers. Aside from modest differ- 
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% -^ccs in cost, itesc dccisiwis should be matk on the basis of policy 
objectives. To which progr^s do managers wish to apply the results? Are 
specific services earmarked for furtl^r development? Is descriptive 
material on the range of all employers' experiences needed? TT^ research 
^ign jdiould be shaped by these decisions: it should no! drive them. 

Any analysis of participating employers must include all participating 
employers within the program services to he analyzed. In the event that 
one participant is plaoKi with more than one employer before program 
termination, both employers must be incluikd. To identify the sample of 
particiixiting employers only fiDm the population of those who were the 
"finar* employers, is to bias the employer sample by eliminating a group 
of placements that worked out especially pcHiriy—lhmc that ended 
prematuivly and were followed by transfer to furtlK^r trtjalmcnt. 

Sample Size 

Only one issue differentiates participant and employer sampling with 
regard to sample size: the completion rate for employer surveys will 
prol^bly reach 80 percent or move. Therefore, the initial sample of 
employers required to produce a target sample of completed interviews 
is smaller than that required with participant samples. For example, if 
we decide to aim for 400 completed interviews and expect an 80 percent 
completion rate, an initial smiple of 5(K) will sufTtcc, For a participant 
survey with 70 percent completion ra'r, the figunr would be 571. 

Intcffratinf^ Employer and Participant Samples 

Combining employer ami participant data is recommended for any but 
the most basic marketing study or descriptive analysis of employer 
benefits. If toth employer aiKl participant follow-up analyses are con- 
ducted, samples should overiap as much as jKMisiblc. The validity of each 
in no way depends on the degree of overlap between the participant and 
employer samples; it depends on the representativeness of each sample. 
A sample of participants selected at random will produce some propor- 
tion with employers — those employed at termination, or those in em- 
ployer-baseo programs, depending on whether the employer survey is of 
termination or participating employers. This sample selection also 
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pnxiuccs a random sample of ihe program *s employers, and insure?; a 
sut^tsmtial number of cases in which bi>lh participant and employer data 
are available for joint analysis. 

Folh^^Up Period 

Employer surveys should allow a lag time after hir;, long enough for 
the employer to observe the former participant's woiic and decide 
wlKJther to retain the individual but short enough l.> allow employers to 
letsiin clear impressions of f^icipants who remained on the job only 
briefly. Lag time should also be shon enough for impressions of woiicing 
with JTPA to remain salient. Since most positions gained by JTPA 
graduates have relatively short probationary periods, a th^re-nwnth 
foUow-up survey «^ould be adequate for an employer survey. This also 
gives participating employers who hire participantsat termination enough 
time to observe their new woiicers under full-wage conditions. In cases 
where OJT positions involve a posttermination performance payment, 
follow-up should occur at least one month after the final perfomiancc 
payment to avoid distortion from expected payments. 

Daia Collection Methods 

Most employers rely heavily on telephone communications and re- 
spond well to telephone interviews, especially if tl^y sac scheduled 
beforehand. Agency personnel who work with employers may resist 
interviewing, feeling that the intrusion on employers* time jeopardizes 
good will. Experience with CETA and JTPA surveys shows, however, 
that brief interviews arc usually accepted and the majority of employers 
are pleased thai JTPA staff care enough about the quality of their program 
to check with those who consume tl^ir products. 

In-House vs. Third^Party tkUa Collection 

Because JTPA staff work closely with many employers, there are 
program development advantages in having staff conduct the interviews. 
These interviews are efilcient because they occur alcmg with other 
employer contacts. They also allow staff to enhance their program 
development and employer quality review by integrating them with 
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employer data collection. StaiT interviews may introduce re.vponse bias 
(HTDbicms, however, because {^icipaiing employers may wish to par- 
ticipate ag4in and may be less than candid about tteir costs and benefits 
from participation. 

Viewed from the standpoint of measurement validity, research efforts 
must be neutral. If the results of an employer analysis arc to be dissemi- 
nated publiciy, both the employers responding to the survey and the 
re^arch consumers must be assured of the neutrality of the measurement 
and analysis, and of the confidentiality of individual responses. Simi- 
larly, if employers tl^mselves are being assessed, a third-party rei^arch 
team ^uld colleirt data, with guarantees of confidentiality.' ' If em- 
ployer surveys are c(nKiuctai in-hwise. elfc»ts can be made to ensure the 
perception of neutrality (see Dillman 1978; Bradbum 1983); however, 
these measures caimot successfully emulate third-party neutrality. 

Es^im&ing Em^ytr Cosls and 

Benefits Using the Gross Impact Approach 

Jc* training programs have an impact on employers as well as on 
partici)»nts. Employers may be viewed as direct beneficiaries of tte job 
training system arni in scnne cases, as incurring costs of providing 
services to that system. In fact, it is often difficult to .separate benefits to 
employers from benefits to participants and society. When a placement 
vioAs well, all benefit. When an employer provides training, the partici- 
pant can ba:ome more employable (eitl^r within the firm. o. generally), 
aiul the employer can gain a more productive woiiier. Similarly, the wage 
subsidy employers receive is rewarding to them and also to the paitici- 
pants, who receive full pay for a period of partially subsidize woit. For 
measurement purposes, employer zmX% and bei^fits may be treated as if 
llwy accrued only to employers. However, interpretaticms of rei^earch 
fuKlings should rwrognize that the most effective sys^ms can probably 
beiKfit all actors — employer and participant benefits need not be mutu- 
ally exclusive. 

Employer outcon^s are not specified in JTPA legislation. Nor is there 
a long tradition of past research focusing on and defining them. Indeed, 
our initial directions in exploring possible measures aie the result of two 
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major limiis lo nK;a.sun?menl and research design in ihin case: the lack 
of prior developmeni in ihis an?a of rejiearch. and the inabiliiy of gross 
impact research to estimate net impacts. 

Employer Estimates of Their Own Casts and Benefits 
Within the limiis of the $nyss impact appmach. the true net impact of 
JTPA on employers cannot estimated. Tliat would require comjwi- 
sonK with employers hiring non-JTPA paiticipanir.. Nevenheless, we can 
ask employers to give us their estimates of their eosts atid benefits. This 
may be accomplished by specifying a break-even point for each measure 
of cost or benefit, md asking employers to repcm w|ielhcr their experi- 
ence with JTPA o * with specific JTPA panicipanis fell above or below 
that point. The break-even piini differs by type of measure and is 
discussed below. The strategy in each case is lo express the measure in 
terms simulating true or perci'iveH net cosi or benefit* by wording the 
meitsure in lemis of brcak-evi n pcint and offering responses on either 
side of that point. 

Although such an approach is far from true nei impact* it may provide 
knowledge of employer outcomes and ho\%^ they combine or offset each 
other. This, in turn, will build a knowledge base require before concep- 
tualizing a net impact analysis. In particulat , we can analyze the degnees 
of associatiOT among different measures of coj t ch* benefit and assess the 
relative importance employers assign to these factors. However* the 
main value of this approach is thai it allows approximations of employer 
costs or benefits useful for guiding program development. 

Measurement Strate^iy in a New Area ofStuay 

As a relatively new area of study, employer benefits cannot be 
measured definitively. It is possible to specify a ran}(e of probable 
benefits and costs, hut too little is known about each or about their 
relative worth lo employer^ to develop a precise, meaningful accounting. 
Some of tlicse costs and benefits, such as the OJT wage suKsidy * can be 
expressed in preci.se monetary terms. Others may be et|ually impo.nant 
to analyze* but impossible to quantify or even to concepiualize clearly . *^ 
They include the following: 
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1 . Th^ major imlicatkMi that hiring a JTPA panicipani was rewanling 
to an employer is a decisiion to retain the |mnicipant. It might also be 
posksible to estimate how far above or below a break**even point (a 
point of iiKJiffeience. neither coHtly nor rewarding) each jwlici- 
pant*$ perfomiance falls; however, Huch estimates would n!main 
speculative. 

2. Fear that JTPA participants may have serious pfx>blcms not easily 
observed before hire is imfwssible to quantify in precise momrtarv 
terms, but is a very important cost for mmiy employers (SimpscHi 
1984b), Even when no problems arise, the perreived risk that they 
might occur represents a cost. 

3. Provision of training is costly to participating employers. Assigning 
quantitative values to employer training, however, is difYicuh be- 
cause most training is informal; more training may be planned than 
occuI^; most training would be offered to all new employees regard- 
less of JTPA involvement; and much of the training may be so 
specific to the particular employer that it binds the worker to that job 
rather than transferring to other employment situations, intrcKlucing 
a hidden benefit. 

Other elements of the JTPA program arc complex to conceptualize 
because they may act as either costs or benefits. For example, employee 
screetting can be a service to employers, but giving partial control over 
screening to an agency whose goal i^ serving the disadvantaged may be 
costly. Similarly, hiring tl^ disadvantaged is typically assumed to be one 
cost to participating employers- Yet, one study of CETA OJT employers 
found thitt over one-tenth of the respondents listed the knowledge that 
'*you arc helping others with need" as the major reason for participating 
in OJT (Simpson 1984b). 

We face these measurcment challenges primarily because little work 
has been completed in this arc a, and many of the most im{K)rtant costs and 
rewanis to employers are inherently perceptual and. therefore, not 
readily susceptible to monetary quantification. The approach suggested 
by grc^s impact analysis is to develop multiple measures of potential 
costs and ber^fits to employers and investigate the extent to which each 
is perceived hy employers to act as a cost or a benefit in their specific 
cases. The following are some examples: 

ER?C 2 i S 
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1 . Ome we leam which aspects of JTPA employers csiimale to be most 
costly and most rewarding and how importam they pensive different 
costs and henefits to he^ analysis can begin to define and prioritize 
employer outcomes, 

2. We can analyze whctter particular types of employers have different 
perceptions of the costs and benefits of JTPA, and whether these 
ideas are associated with greater or lesser program succe«i for par- 
ticipants. 

3. We can test ideas about the ways in which JTPA is rewarding or 
costly to employers. Rather than assuming that f^irlicular JTPA 
services, such as client screenings are cosily w rewarding to employ- 
ers, we can examine the extent to which the implementaticMi of these 
services increases or decreases the rewards or costs perceived by 
employers. 

4. We can analyze the association among different measures of cost 
and benefit. Are costs of providing vScrvices higher where benefits, 
such as the subsidy to participants' wages, are higher? lk> employee 
who receive high levels of one type of benefit tend to receive less of 
others, or is JTPA implementation such that some agencies reward 
employers dcross-the-board more than others do? 

Outco^s /or Termination Emphyers 

Any employer's major costs or benefits from hiring are the job 
performance qualities of the new employee. For termination employers 
who did not participate in delivering JTPA services, this is tte only 
souree of cost or benefit relevant to JTPA. The question for them is 
whether the new JTPA-train^ employee will function in the job as weU 
as other appropriately trained w workers. There is no reason to expect 
jTPA participants to he better trained than others; the goil of JTPA is 
to eliminate participants' previous deficits. 

Each area of woricer performance of importance to the employer 
represents one dimension, or scale, of cost or benefit. How many days' 
woric will tite new employee miss during the first month? How much 
employer training will be recuired before the worker becomes produc- 
tive? How much supervision time will be saved by a **self-starting'' 
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worker? Cost and benefit represent two ends of each scale. The break- 
even point lies zi the point on the scale that repiesents the average new 
(non-JTPA) hire for that job in that labor market, as perceived by the 
employer. If the average new-hiie misses four days' work per month, 
hiring a former JTPA (^icipant who misses an average of two days 
represents a benefit to tl^ employer. 

For employee qualities that an? not naturally quantiHed, the former 
JTPA participant may be compared with the average non-JTPA hire for 
that san^ job, using a rating scale such an "much better/' "a little better/' 
'*aboui the same/* "a little worse/' "much worse/* Qualities to measuie 
include skill level, speed and quality of work pnxluctiviiy, indicators of 
supervisability, and indicators of adiiistment to the job. 

In the case of my one participant, job performance may be better or 
worse than average for reasons unrelated to JTPA participation or 
referral. However, if, over a large number of employer interviews, the 
average JTPA hire proves to be more satisfactoiy to employers than their 
average non-JTPA hires, we have reason to claim a role for JTPA in 
producing that benefit to employers 

A second possible benefit to termination employei>i is a former 
participant's job retention record thitiughout the follow-up period, and 
whether further retention is likely. Retention implies that tl^ worker is 
productive and ^jiLsted, and also wishes to remain employed. Unless job 
loss results from cutbacks forced by declining business, laying ofT a 
trained worker indicates a cost to the employer: the cost of hiring and 
retraining another worker, and loss of productivity during the training 
period. (See Vermeulen and Hudson- Wilson 1 98 L ) Whether these costs 
occur because participants perform poorly or because they quh is also 
valtiable to explore. 

Outcomes far Partk^^ng Empioyers 

An employee's productivity and Iractability during the training con- 
trol represent major costs or benefits to employer* who take part in the 
JTPA program. After tl^ contract, they may become termination em- 
ployei^ by hiring tlKJ i^icipant tl^y trained. At this point all the benefits 
discussed above apply. Beyond ttese, the most obvious beiwfit is the 
subsidy to participant waives. 
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By far the mosi common reason employers give for participating in 
OJT is the subsidy. Other explanations commonly reportol include 
eliminating the need to screen large numbers of applicants, the ability to 
expand or to stabilize without mounting the full cost for the new 
employee, and satisfaction at being able to assist deserving individuals 
(Simpson 1 9X4b). Commonly iejx>ned costs include the time and super- 
vision required to train, the potential of greater-than-avcragc worktimc 
lOKt to personal or family prciblems, the possibility that maximum 
performance after tinning will not match that of other employc^s^ and 
the possibility that jTPA employees might turn over faster than others 
would. 

One element of employer costs has declined dramatically since early 
CEl'A pn^grams: the ckgree of constraint exfH^rienced by the empluyrr 
O'NeiPs ( \ 9K2) analysis of employer hesitance to use **targcted job 
crcdits" demonstrates that the sheer fact of being constrained cai. 
costly to employers. Earlier CETA programs protected their tight to 
serve participants with greatest need* but in so doing, raised the employer 
constraini expenses above the threshold allowing participation. 

Some of these costs or benefits have break-even points of /cm. For 
example, the OJT wage subsidy cannot be costly in and of itself, and 
paperwork requirements cannot be seen as benefits: they can at best pose 
zero cost* Other benefits and costs to participating employers are 
meaningful only when a break-even point is defmed in a>mparison to 
typical employees who would be hired were it not for the JTPA program. 
The two major outlays JTPA wishes participating employers to accept 
arc hiring individuals who appear to be less qualified for the job than 
typical non-JTPA hires, and providing extra training beyond that re- 
quired by typical non-JTPA hires. The issue is not, for example, wl^ther 
the OJT employer loses five or 10 weeks of productive time during 
training, but whether the difference in training rime for typical non-OJT 
hires vs. the OJT hire is zero, five, or 10 weeks, A difference of zero 
weeks represents a break -even point on that particular measure. 

There is no a priori method to establish a balance between major costs 
and benefits for participating employers. Program policymakers must 
decide whether they are satisfied with the differences employers rejwrt 
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between JTPA participants and other hires, given the JTPA aMmbursc- 
menl they receive, l^ta on employer benefits imd costs simply make 
decisions such as ^^tting the level of OJT reimbursement more rational 
Even so, such interpn?iaiions may be Ic^s obvious than many program 
managers suppose. In particular, it is not always the case that employer- 
based programs work best for participants when employers receive 
maximum benefit from participation, as demonstrated by economic 
theory regarding nonsubsidizcd on-the-job training (Maranto and RcHjg- 
ers 1984: Hoffman 1981). Employers always engage in introductory 
OJT. specific to the firm and to the job. This training represents part of 
the employer s investment in hiring any new employee. The typical 
sequence is hire with intention to retain, invest in training, and retain as 
planned. 

ITie subsidized OJT situation differs from this typical sequence in that 
the traininfi (k cws before the dei -ision to retain, and the /w//;//i>f may m t 
he the result of a decision to invest in training. If the total cost of iraini ig 
a JTPA participant is greater than the income derived from the v^age 
subsidy, the employer must decide to invest in training, which in turn 
implies a commitment to hia* if possible, so as not to wasle the 
investment. If, however, an SDA offers srbsidies equal to or larger than 
the employer\s cost, the employer may pii ticipate without ever having 
decided to invest in the participant. The reason may be kind — "Now I can 
afford to help this jXTSon." Or, it may be hard-nosed — "I make moa* 
money hiring OJTs, even if i increase turnover by letting them go after 
the contracl ends/* 

At the extreme, some participating employers use the federal wage 
subsidy without incurring the expense of providing any services. In- 
depth interviews with CETA OJT employers located some who explic- 
itly staled that itey provided no training and refused to alter their hiring 
practices at all, choosing instead simply to gather the windfall wherever 
one of their new hires happened to be OJT eligible (Simpson 1984b). 
Therefore, service ]m>viders are presumably well-advised to balance 
costs and benefits for participating employers in such a way that 
outcomes arc positive, but not so positive as to protect employers from 
making an investment in each participant ihey train. 
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Outcomes Measured through Agency Records 

The most basic benefits accruing to participating employers are linan- 
cial, and may be recorded directly from JTPA contracts. TTtis form of 
measuiemcni is preferred because it is highly reliable, it indicates both 
planiKx! and actual expeiidiiuties, and it avoids tlK^ awkwardness of 
asking about money during telephone interviews. In addition, agencies 
may be able to estimate the amount of screening and refeira! time they 
provided, thereby offsetting employer hiring costs. The question of how 
effective the screening was is separate, ami must be measured during 
employer interviews. 

Outcomes Measured through Employer Surx eys 

In addition to measures listed earlier for termination employers, par- 
ticipating employers incur a number of costs during their contracts, and 
also experience the potential costs and benefits of working with JTPA 
agencies. Tliese may be measured through follow-ups, in the form of 
employer reports of their activities or perceptions of JTPA. Presumably 
tte most baKic costs incurred in the case of OJT arc training costs. 
Although small employers seldom estimate training costs, they can 
report length of typical training for a given position, length and intensity 
of JTPA training compaied to non-JTPA u^ining, whether specialized 
methods* curricula, or pK^rsonncI are used during training, etc. 

Measures of perceived participation risk. Participating employers 
face the costs of accepting risk or ctmstraint from their involvement with 
JTPA. Although particular JTPA particij^ts may prove to be ideal 
workers, a program offering subsidies in exchan^ for hiring particular 
individuals has some implieu risk. Employers may fear that the employe 
could be a poor woiker, an alcoholic, or a thief. This felt risk may lcx>m 
larger than tlie actual costs expcrictKcd when a particular worker 
performs poorly. The JTPA agency could also attempt to ccHistrain tte 
employer's behavior, or unexpected paperwork demands could develop. 
These possihiiities may be expensive in employers* perceptions. 

At the other extreme, the employer could reduce risk by retaining 
control over the hiring process. In the most extreme case, employers 
make firm hiring decisions before seiKiing their new employees to apply 
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for il^ OJT suteiidy. Since this practice undercuts the hiring-incentivc 
role of OJT wage reimbursement, employer reports may be somewhat 
biased, depending on guarantees of confidentiality. At a more interme- 
diate level, JTPA staff, rccogni/Jn^ employers* fear of risk, may screen 
so carefully that the risk factor is neutralized. Thus, valuable knowledge 
niay be gained from measunes of employers' "felt-risk," employers' 
control over screening, and the degree to which JTPA staff screen out 
paiticii^ts least job-ready (and therefore most in need). 

Measures of employers' direct assessment tff partivipatim costs and 
benefits. Employers may also be asked for their direct assessments of the 
cc^ or bei^fitsof participating. Most of tl% measures suggested thus far 
have been indirect, in that they ask employers to rate a particular JTPA 
participant or placement experience. This approach has the value of 
defusing employer concerns about being evaluated, i.e. . clarifying that it 
is the employee who is being evaluated, and it allows aggregation of quite 
precise information regarding a representative sample of participants. 

Cenain employer outcomes, moreover, are best estimated in a direct 
fomi. Employers can be asked to evaluate JTPA services and staff, and 
to indicate how berajficial or cc«?tly they found sp^ific aspects of 
paiticipaticm to be. Some measures, such as the sul»idy to wages, Mp 
in enlarging or stabilizing the work force, or the good feeling of helping 
others, can logically represent only some degree of benefit. Others, such 
as JTPA applicants' screening, may represent costs or benefits, depend- 
ing on their quality." 

The Characterization of the Employment 
Establishment and of the Participant's Position within it 

Wlwn one gt»l of employer surveys is to perform differential impact 
analysis of participant outcomes, employment-establishment character- 
istics and i»rticipant-selection-aiKl-training characteristics should be 
included among employer measures. The following three levels of 
measurement specificity are encountered: 

1 . Measures characterizing the entire employment establishment, such 
as number of employees, industrial .sector, or referral patterns 
established with JTPA. 
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2. Measures applying to any employee with the same job held by the 
JTPA participant, such as job complexity, qualifications required for 
lhai job, or training level of typicjil non-JTPA hires. 

3. Measures applying specifically to each JTPA participant, such as the 
length of training rcccivcd. or employer's ratings of that participant. 

In cases where SDAs envision repeated liKal employer follow-ups, 
efficiency can be increased and nuisance to employers decreased by 
treating categories 1 and 2 as oncx*-only measures analogous to those for 
service providers. Category I measures could h: taken during an initial 
work-up with each new employer. Category 2 measures would be 
gathered onc^ for each separate job title into which each employer 
accepted JTPA participants. Such measures arc easily integrated into 
program operation where employer or participant analyses arc envi- 
sioned. They can be combined with participant data for analysis as long 
as both employer and participant data include an identifier for each 
employer. 

An Illustration of Differential Impact 
Analysis Including Employer Data 

A summary of selected findings fn>m the Washington State CETA 
OJT study discussed earlier will serve to illustrate the application of 
differential impact arialysis (Simpson 1984a). That study analyzed data 
from a nine-month follow-up of 881 OJT participants and 517 OJT 
employers who trained them. In addition, data from participant MIS files, 
slate labtjr and industry sources, and surveys with all OJT service 
providers in the CETA system were combined with data from the two 
follow-up surveys. Selected findings relevant to one program develop- 
ment issue — quality control over OJT placements — are summarized 
here. 

There is a continuing question concerning the extent to which OJT 
represents a training intervention, with employers reimbursed for addi- 
tional training costs demanded by their pn>gram participation, or a hiring 
incentive program in which employers provide little service except to 
hire from the list of eligibles. In addition, during the period immediately 
preceding the 1982-83 data collection for this analysis, the Slate of 
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Washington had decided to mther dramatically cxjiKind its OJT pmgnim. 
raising the question of whether pressuic to enlarge the pool of new 
employers led to a deterioration in the quality of OJT placements. 

While gross impact analysis canmit assess the net impaci of OJT for 
participants, we were able to test a series of propositions comparing 
fomis of OJT implementation that placed greater or lesser emphasis on 
quality control. We were also to compan? specific OJT placements that 
af^ared to provide service to participants of greater or lesser quality. 
The following summarizes findings from dilTen^ntial impact analysis of 
these implementation and treatment variants on postprogram outcomes. 
All of these reports arc for the outcome variable most clearly affected by 
pn^grammatic variables — whether or not one retained employment with 
the original OJT employer J*^ 

1 . Various qualities of OJT implementation and treatment explain far 
more variation in outcomes than do the full set of participant 
t^ckgmund characteristics included in MIS files and augmented by 
measures on the follow-up survey, although age and employment 
history have considerable impact. This was true in part, we learned, 
because so many OJT positions werc entry level, making such 
minimal ckrmands that some highly qualified OJT enrollees left 
voluntarily, thus undemiining program success. 

2. Service provickr measures indicating the degree to which strong 
quality control prcKedures were a part of their OJT pn^gram implem- 
entation proved strongly associated with the rate at which partici- 
pants retained their OJT positions through the nine-month follow- 
up. A number of factors; raised OJT retention: more demanding 
quality review for new and old employers, a policy demanding 
higher than minimum wage for OJT placements, and a willingness 
to hold some money back because an insufficient number of satisfy- 
tory employers were available for OJT. We estimated a 28 peaent 
difference in OJT retention rates, above and beyond other factors, 
between agencies placing most and least emphasis on quality control . 

3. Consistent with the interpretation that OJT in this system was 
suffering from low quality control, a set of measures designed io 
measure the quality of participant training also indicated higher OJT 
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retention where n:Bining had been more intensive. In particular, 
employei^ who reponed offering any .special instruction for OJT 
I^icipantss who u%&i more formal preparaticm along with informal 
OJT, and who h^ more personal involvement in the training 
irtaiiKd tlKrir p^icipants more often. TTie impact of these factors 
was modest b<^ause training intensity overlaps with two other 
powerful predictors of retention: the complexity of the job, and the 
panicifKint*s enthusiasm for ihc work. 

4. While it is common to worry about OJT participants' ability to meet 
their job demands, tte reverse proved much more problematic in this 
study. The less background the paiticipani reix)rted having in the 
area of the OJT job and tte more complex tte employer described the 
job as being, the higher the OJT retention (after adjustments for 
participant background characteristics). We foutid that only 7 per- 
cent of the participants were fired for inability to do the work, while 
10.3 percent gave boredom with the job or getting no training as ite 
main reason they quit, and another 14 i^rccnt left for a better job. In 
all, 31 percent quit, while 2 IJ percent were fired or left by mutual 
agreement with the employer,*'' 

5, The three strongest predictors of OJT retention were the employer's 
rating of how enthusiastic and cooperative the particij^nt was, the 
employer's rating of how fast ^ participant worked, and the parltci- 
pant's felt importance of retaining a carrer in the type of work rep- 
resentcxl by the OJT job. All these turn out to be much higher when 
the job is more complex, when OJT positions provide more training, 
when particij^ts are moving into a new area of work rather thim 
being placed in a job about which they know a great deal when 
employers more frequently provide evaluative feedback to pj^-tici- 
pants, and where service providers emphasize quality control. The 
higher the quality of the OJT placement, the more likely iwticipants 
were to like the job and treat it as a career they valued, and in turn 
display behaviors employers wanted to see. 

These findings conclude that in that particular OJT system at that time 
the program needed to develop quality control over the nature of the OJT 
site, i.e„ ite .services offered by employers. One other ux>ublesome 
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finding consistent with this concern for quality was the discovery that 
some employers in that sample were explicitly using the OJT reimbunic- 
ment as windfall profit. 

Although employer-initiated OJTs were quite common in this sample 
(45.8 perce.,* of all OJTs), most of tlK^se represented referrals by 
knowledgeable employers who made no hiring dix^ision until eligibility 
was established, Postpn^aram success in these cases was no higter than 
average. However, one-sixth ( 17.3 pei\:ent) of the employers we inter- 
viewed said they first maikr a fmn hiring decision and then 5^nt the 
participant to CETA to sec if a wage subsidy could be gained/"*' This 
phenomenon rcp.-escnts both a poor expenditure of training dollars and 
a selection mechanism likely to bias outcome estimates. Among these 
participants, retention was 12 percent higher, after adjusting for other 
factors. 

This set of findings was chosen to illustrate the value of differential 
impact analysis because many separate tests lead consistently to tii^ same 
conclusion, and because that ctMiclusion is in essenc*e opposite to the 
normal interpretation of weak program perfomiancc. When one service 
provider perfonns at a higher (descriptive) rate than another, nearly all 
analysts will ask whether that difference was produced by "creaming/' 
i.e,. whether the finding jrpresents selection bias. Few will ask, however, 
whether participants were too highly qualified, relative to the quality of 
the OJT jobs and training. Yet, careful quantitative analysis of program 
implementation and treatment confirms the latter interpretation in ihis 
one service delivery system. 

How Gross Impact Analysis 

Complements Net Imfwct and Process Evaluations 

The most valuable uses of the gross impact evaluation meihod — and 
also its major limitations — may be placed in relief by a brief examination 
of the ways in which the three approaches in this volume complement 
each other The gross impact approach exploits its measurement flexibil- 
ity to enlarge the range of outcomes analyzed as well ^s the range of 
fmrtors considered as imluences on outcomes, yet the quantitative nature 
of its measures helps insure that coiKiusions are reliable. 
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In the Washington State CirTA OJT example, the gross impact 
analysis provided no knowle<ice of the net value of that OJT system to 
its panicipanis. If we had employed both the net and the gaiss impact 
approaches, the major value of ihc gaiss impact findings would have 
been to broaden the range of measuivs analyzed — both outcomes and 
programmatic factors ihai might have influenced the net impaci. These 
nieasurcs would have imrvased our ahiUty to explain how rhe system 
works to pnnJuce rhe hi^h or hw net iwfhivis we identified, pmviding 
guidance on improving net impact. Without a net impact evaluation, we 
do not know how urgent the need for system improvement is. Without the 
gross impact evaluation, we have less guidanc: regarding the mecha- 
nisms needed to improve a system,^^ 

At the other end of the continuum lies pnKcss analysis. Its detailed 
analysis of program implementation feeds gross impact analysis by 
identifying measures worthy of quantification. Only when the analyst 
understands the process by which organizations oi^rate will meaningful 
outcomes he selected for measurement, or will the analysis of program- 
variant effects upon outcomes be meaningful. The centrality of implem- 
entation factors to differential impact analysis means that gn\ss impact 
analysis is in part quantified prcxress analysis and should always be 
preceded by at least a partial pnxress analysis. 

In addition to its IVxrus on 'outcomes, gross impact analysis adds four 
major complements to pnx:ess analysis: {\) pt stprofiram outcc«ne 
measures, (2) quantitative precision of measurement, which allows 
rt^liable estimates of the impact of program variants on outcomes. (3) 
reliable c<nnparisons acn>ss multiple service providers or treatments, 
and (4) measurement of mdividuul service treatments as possible influ- 
ences on outcomes, along with measurement of the implementation 
factors also emphasized by prcvess analysis. 

Because process evaluation avoids the limits of formal, quantitative 
data collection, it can be flexible, creative, and unique. However, for the 
same reasi>n, i.e., process analysis does not collect quantitative data, 
conclusions from process analysis are subject to considerable error, 
which can be rcduc^ by subjecting the conclusions of process evalu- 
ations to a gross impact analysis. Such interpanations can be tested in 



Evaluatinp Gniiss I^n^gmni Ouiaimcs 221 

multiple contcxtH and can include a range of measunis indicating individ- 
ual treatment as well as program implementation. Hius, lor example, a 
proa!S!s evaluation conclusion that an orgmii/iition is less elTective than 
it might be because of some particular element in its internal struct urc can 
be tested by comparing outcomes, as indicators of effectiveness, among 
organizations differing with regard to that structure. 

The CETA OJT study, discussed in order to illustrate diflerential 
impact analysis, began its conccptuali/iition phase with an informal 
process analysis. Our aim was to identify alternative thci>ries regarding 
OJT system operation, pn>gram variants likely to impact outcomes, and 
program variant n ihat cH>uld be changed if policymaken* decided to use 
our findings \y improve program performance While siandard variables 
for such rcs*;arch were also measured, several analyses that proved lo be 
most fasciniiing mvolvcd variables thai emerge J from the pnKcss 
analysis. Had the research ended at the pnxrcss analysis stage, these ideas 
cou?d iK)l have vKcn tested and could no! have generated quanti;:«tive 
estimates of the imf'-ict which program variants can have on outcomes. 

Chapter Summary 

Tlie major goal of thegrt>ss impact evaluation approach presented here 
has been to improve the technology available to managers of human 
services organizations. Technology, in this case, is knowing how to 
operate programs that effectively produce the desired outcomes — tnms- 
forming clients with given needs at intake into postprogram success 
stories. Gross impact analysis approaches that goal with two distinct 
analytic sti^tegics: the analysis of descriptive gross outcomes, and 
differential impact analysis* These analyses are performed using data 
fitwn several possible sources: MIS files, participant interviews, data 
from scr\'ice providers on individual treatments and program implemen- 
tation, and data from others, such as employers, who may be closely 
involved in the operation or outcomes of the program. 

The analysis of descriptive gross ouUromes is useful because it is 
simple. It is also dangerous, for the same reason. The following steps may 
be taken to enhance the usefulness of descriptive outcome measures: ( I ) 
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broadening the range of outctnne«i studied, to provide fuller interpreta- 
tions of how the program h funclioning; (2) tailoring measures and 
analy!iiii quesiiom m n^iuke findings more meaningful; (3) measuring 
views of those whose pea^ptions have prima facia meaning (eg., 
employers); and (4) insuring high technical standards during data collec- 
tion and analysis. Analysts can also protect against misuse of descriptive 
reports by limiting the nature of their interpretations to those merited, 
given the limitations of the analysis. 

Hie second analytic approach, differentia] impact analysis, is more 
expcasive and also more useful for program itevelopmenl. This method 
increases the level of technology available to managers by testing the 
relative effectiveness of alternative forms of program implementation or 
servici? treatments provided to individuals. That is, differential impact 
analysis estimates the degree to which difference in postprogram out- 
comes is caused by any given pragram variant. Each alternative is tested 
against other alternatives which are in place in the service deliver>' 
system. 

Further, differential impact analysis aUack.s the problem of selection 
bias, the primary factor limiting the validity of descriptive gross outcome 
repons and inhibiting causal inteq>retations. By using a variety of 
sources to measure selection processes as well as client characteristics 
predictive of postprogram success, the most powerful ^^alternative expla- 
nation'* facing all program evaluation — that outcomes were produced by 
client characteristics t)r selection rather than by the program— can be 
greatly mitigated, if never completely eliminated. 

Thoughtful preparation of survey data coHctrtion \ooh and appropriate 
use of analysis techniques, available in a wide range of statistical 
pmrkages, can make differential impact analysis a powerful tool for 
improving program performance on mandated outcon^s. At the same 
time, the wide spectrum of measurable and describable gross outcomes 
can be used to improve or maintain the quality of service while core 
outcomes are being maximized. Findings apply directly to the state or 
local service delivery systems in which the evaluation was conducted, 
directing managers to increase some s *rviccs or retain some implemen- 
tation forms, while reducing others. Local applicability of findings. 
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protection against selection bias, and testing of alternative program 
variants provicte the basic component!? of quasi-experimental causal 
analysis, so lhai careful difTerential impact analysis produces results that 
can more accurately identify factors influencing program clTectivciKss. 
Changing these factors is. therefore, likely to produce improvement in 
effectiveness. 

NOTES 



Cm^s impati analytii^ kkmiftrs whkh services Urticr for cavh gn^up. 

2. The ranite of avaiteWe ahemalive enplasvuium may be itkntilVed hy askii^ manafsm itf 
programs thm are perftHining pinnly to cxptetn iheir cv^ani/^knus' ^tak i^lHYWing. 

3. The aim of all n^M^h is to inctettse or n^tiKrv nut cimfufcnce in particular comrlustcHis tif 
inbrrpiri^iom. To reject new infimrmum hecau.se it is imperfecl is 5»s fciottsh as to etnbr^T 
unidiaWc findiitgs wMcheoncdly. The Bs^ysi must assess ite *ali» t;>f any le^iffch fimJin|i. 
Sinictimng the reseaah so ihai fmdings are fimter impnives the valijc of rei^arLh, even if ihe 
method trmains ciwKidcr^l v less than perft^t. 

4. Since ttesc (t%uU% m deschf^ive only, this interpretaiK«i is also subject toemn. il could be. 
for example, that mi>re AFDC lecipiems k?ave their OTT posit kms because they arc unstable, while 
non-AHX" recifHcnts leave in nnier to move to high^ paying Jti mmfetatcd areas. Such 
posKihi*' :m he ie«ed if the reseaaher has enlcnained Ihrm c^ly enough to make the data 
ava* ihis case, fitr example, msre mm- AFIX? did quii for hcner >>!«s but rm enough 
mm »MvaIidate the iniiia! imerptrtmitm offeied in the le^l. 

5. This ff guir mKl iHhers m this series may be somesvte lower than lyp^cal since Washington was 
exper^ncing lather serious recessttm during this stmty. 

6. ^>ve and beyond the obvit»us reason that iTPA now requires such a tk'finttitm. 

7. Oiw ti>pic of p^tcular imetest lo smaJI SDAs involves the correction (k>wn\%'aKi of ibe sampJe 
nee<^ for a given error margin when xhe popuWuns from which the sample is drawn is very small. 

K. This docs suggest iwe pcm.sihle pitfalJ of tow- budget pn>gram evaluatitm: if surveys are 
a]9tduc^ in')^7use4 nr%pc»idenis may bias their answers in a positive directimt, Sophisiicated 
extenul consumers will therefc^ temJ lo qucMiim findings Iwed <mi surveys, unless they are 
conducted hy third pariks md guarantee ci^Adentiality. 

9. These membership icfemifa^rs alM» serve as extivmcly import am coninii variables under cchi- 
dhksns djscus.sed later in the chapter. 

10. The term ^'membership akniifier** has been used thrmtglKuit this clapter to refer to what 
suuisticians typkally call "dunmty variables " The ^imdsaiH rules governing prc^ analysis of 
dammy vaotibles should be f<^lowed during the analysis desiribcd here. If all membenAip ^kmi- 
f»rs are entered •limidtaneously, cmly ad rnay be incltKted. Fw example, if 15 sen ice i»ovidersare 
included in the sample. mefl^ier«Aip identifttfrs idummy variables ) indicating 1 4 may be included in 
a single regresMon equation. If the analysis involves a forward stepwise (»ocedufe. n lalU 
men^)crship identifiers may be includird. 
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H . ll shmhi he m>i«;d ihal because interAik^ tenrn an.' highly amvlatcti wt!h their cnnMitiarnt 
vnriableii. analytOs <JKHikl itimuh ib^ chimge in vaiiancr explained (Ri) raihcr than a-King on i or 
F MX}nes far each imlivkluitl variable within the mcnkl. 

12. When |^>fTams aie nn»l smrcesiiful. neariy all jwtkipants aie cmplcmrd sifter the fmigram. 
Variance in earnings is gresOeM wten a lai^e pmpi>rti4?n of rarticipiinis cams mHhing. 

1 3. Opiitmlly. a written kmn with ar^wymcHis return amkl he mhninisteanl by <4aft. hut ma 
wiihiiui itu' jm>Mcms of ivtum n^te ami > etariul iivcrlt^^ which acctmip^v mail Mir>'cyi». 

14. Examples of such qu^-Mitms as ih^y appear in a sur\ ev aa- avaiJahk in Hinipscn ( 1 VHb). 

1 5. Ft>r a set (vf spet ific suggesitnf measures ih;H have been pte-tesied umt^ig JTPA employers, see 
SttnpsiKi, I9Kh, 

J 6. One difficult)- w iih survey data emerges hen?. Respc mtlents often lend to bias tteir repwts in 
a positive diieiiion. so thai the midpi»int of any sel of answer* is always a hii above the face validity 
mtdpoini , Thus, if a git^p of cmpkiyers v^vrc asked to mie all tlwtr employees, the average emplo)^ 
wi>uld be rated stimewhai ohove average. Analysts shtmid bear this in mind wl«;n inieipreting 
enipkwr ratings. Iit>wever. m> preci;*^' information exists with «hich to estimate to what extent 
respimj^s are inflated, niaking ;KJ)ustmcnts iinprecise. 

1 7. Sec SimpM^ { 1 9X6) for specific measuren^nl suggestions fwlhis ivjiue as for other employer 
issues. 

1 8. One of our nioie intriguing fimiings uas that the ability lo rcgaui other cmployir^it imx tlx* 
OJT position nas lost was alnnjst enUrelv immune lo inlerpretation via prxigram variables. Tfeii is. 
the impact of the OJT prngnun- <n, rather, any varialttHis in iis implementation^ -extemied tmly to 
getting and retaining tl^ original OJT H^b. 

1 9. For valid measurement of issues such as this om\ it provejt especially useful lo have data fn>m 
both the empbyer and the participant. 

20. Pivsumahly. employers ntrre candid with uv because w-c wre & neuind third party , S4>me 
employers even offered explicit statements thai they view«l the entire pnxTss, cynically, as a 
windfall. 

2 1 . h is possibte in theory . and at great expense, to conduct net impsKrt evaJuatk>iwi it*al ii^lude a 
broad range of outtx^nte measures. TTje uxst-efTicienl design suggei^ed in this volume by Jt^nson 
fortrgws this possibility to maie ibc teseart h feasible for stales and large SDAv. Any quantitative 
analysis of a wide range of pn»gram vidants autonnitical t> bettmres a gtx>ss imt^ analysis because 
these measures are meaningful only among ponicipanls. An unnvated ct^nparistm gn^up must, 
therefore, be omitted fmm si^h an analysis. 



Evaluating CJross Pn>grani Ouicomes 225 



RKFERENCES 

Anlin. Alexander. Four Critival Years: Effects ofCoUeae on Beliefs. Attitudes, 

and Kmpwleiige. San FranciMro: JoHHey-Bass. 1977. 
Babbie. Earl R. Survey Resean h Metlukis. 5lh ed. Belmont. CA: Wadhworth 

Publishing Co., 1989. 
Bjomstadt Get^rgc W., and David Knf^c. Statistics for StKial Data Afiulysis. 

Itasca. IL: F. E. Peacock Publishers. Inc.. 1982. 
Blaiock. Hubert M. Causal Inferences in Nimexperimeptal Research. New 

York: W. W. Norton and Co., 1964. 
Bialoek, Hubert M. Causal Models in the Sjcial ScienccA Rev. cd, Chicago; 

Aldine, 1985. 

Blalock, Hubert M.*'Theory Buildingand the Statistical Ccncept of Interaction.** 
American SiH iological Review 30 (1965): 374-380. 

Bloom, Howanl S., and Maureen A. McLaughlin. C£T/4 Training Proiirams— 
Do They Work f(}r Adults '/ Government report by the Congressional Budget 
Office and the National Commissicm for Employment Policy ( 1982). 

Bonis, Michael, and William Tash. Measnrin}^ the Impact of Manptm^r Pro- 
grams: A Primer. Ann Aii>or. Ml: institute of l^abor and industrial 
Relations. Tlte University of Michigan-Wayne State University, 1979. 

Bradbum. Norman M. "Response Effects," In Handb<H)k ofSttn ey Research, 
edited by Peter Rossi, James D. Wright, and Andy B. Anderson. New York: 
Academic Press. 1983. 

Campbell. IXmald T., and Thomas Cook. Qmsi-Kxperimentati(m: Design and 
Analysis Issues for Field Svtnngs,Chkai^o: Rand McNally. 1979. 

Campbell. Donald T., and Julian C. Stanley. Experimental and Quasi-Experi- 
mental Designs fi>r Research. Ch\c^2/^\ RaiKl McNally. 1966. 

Caporaso. James A., and Leslie L. Roos. Quasi-Experimental Appnmches: 
Testing; Theory and Evaluating Policy, Evanston. IL: Northwestern Univer- 
sity Press. 1973. 

Diliman, Don A. Mail and Telephone Surveys: The Total Design Methrni. New 
York: John Wiley & Sons. 1978. 

Doeringer, Peter B.. and Michael J. Piore. Intermd Lahor Markets and Man- 
power Analysis, Lexington. MA: D. C. Heath. 1971. 

Fink, Ariene. and Jmrqueline Kosecoff. How toCondttct Surveys: A Step-hy-Step 
Gidde. Newbury Park.CA: Sage, 1985. 

Franklin, Grace, and Randall B. Ripley. r£T/»; Politics and Policy. l97Jt'I9S2. 
Ktioxvilk. TN: The University of Tennessee Press. 1984. 

Frey. James H.Suncy Research hyTelephtme. Newbury Park. CA: Sage. 1983. 

Gentci. Vincent J. Short Term Indicators of Job Training Program Effects on 
Long-Term Participant pMrnings. U.S. Department of Labor. August 1 984. 



ERLC 



226 Evaluating StKial Problems 



Gillespi. Michael W. "Log-linear Techniques and the Regression Analysis of 

Dummy Dependent Variables: Further Bases fwComparist>n.** Sm iolo^i- 

cal Methods and Resean h 6 ( 1977): 103-122. 
Goodman, Leo A. "A Modified Multiple Regression Apprwich to the Analysis 

of Dich(»onious Variables." American Sotufio^k^l Review 3H\972): 28- 

46. 

Hoffman. Saul D. "On the Job Training: Differences by Race and Sex." Monthly 

Labor Review 1 tM ( 1 98 1 ): 34-37. 
Ki.sh. Leslie. Sim'ey Sampling. New Yoik: John Wiley & Sons. 1%5. 
Knoke, D. "A Cranparison of Log-linear and Regression Mtxlels for Systems of 

Dicholomous Variables." Sm ivlogival Methods and Resean h 3 (1975): 

416-434. 

Levitan, Sar A., and Garth L Mangum. Tlte T in CETA: Lwal and Natitmal 
Perspectives. Kalamazoo. Ml: W. E Upjohn Institute for Employment 
Research, 1981. 

Maranio, Cheryl L, and Robert C. Rodgers. "Does Work Experience Incrca.sc 

Productivity? A Test of tteOn-the-Job Training Hypothesis." The Journal 

of Human Resounes 1 9 ( 1 984 ): .34 1 -357. 
Minnesota Office of Statewide CETA Coordination. Private Sector Experience 

with CETA On-the-J(?h Training. St. Paul. MN: Minnesota Departn^nl of 

Economic Security, August 1979. 
O'Neill, Dave. "Employment Tax Oedit Programs: The EfTecLs of Socioectv 

nomic Targeting Provisions." The Journal of Human Resources 1 7 ( 1 982): 

449-459. 

Rossi, Peter H., and Howard E. Fn.»eman. Evaluation: A Systematic Appntach. 
3d ed. Beverly Hills, CA: Sage, 1982. 

Rossi, Peter, James D. Wright. atMl Andy B. Am erswi. Handhtwk of Survey- 
Research. New York: Academic Press, 1983. 

Seattle-King County Private Industry Council and Sncdcker Scientific, Inc. 
JTPA Evaluation Issues. Priorities, and Contingencies at the SDA Level. 
Technical P ,-port for the JTPA Evaluation Design Project, Washington 
Slate Employment Security Dejwrtn^t, 1985. 

SimpstMi, Carl. Vocational Classroom Training Outcomes Project: Final Re- 
port. Olympia, WA: Washington State Employment Security Department. 
1982. 

Simpson, Cari. CET 4 On-the-Joh Training Study Final Report. Olympia, WA: 
Washington State EmployrtKnt Security Department, 1984a. 

Simpson, Carl. CET A On-the-Joh Training Employers: A Surwy of Employers 
and OJT Officers. Olympia, WA: Washington State Employment Security 
Department, 1984b. 



ERIC 



FAaiuating Gn^s Program Outcomes 227 



Simpson, Carl. A Guide for Gross Imimvt EvaiHaUons. Olympia, WA: JTPA 
Evaluation I^ign Project* Wa^ington State Employment Security De-^ 
partment, 1986. 

Simpson^ Cari. '^Cwnj^teney-Based Assessment and Ren^iating Barriers to 
Employment: A Quality Analysis of Ow SDA/* Evaiuatian Forum 4 
(1989); 24-26. 

Snedeken Bonnie, and David Snedccker. CEl'A: Dei entrallzaUonon Trial. Salt 

Lake City, UT: Oiympus Publishing. 1973, 
Sufbnan, Seymour. Applied Sampling, New York: Ac^emic Press, 1976, 
Taggart, Robert. A Fisherman's Guide: Ah Assessment of Training and 

Remediauon Strategies. Kalama/oo, MI: W. E. Upjdm lastitute for 

Employment Reseanrh, 198K 
Veraieulen. Bruce, ami Susan Hudson-Wilson. '*The Imj^i of Workplace 

Pmctices crs Education and Training Policy/* In Jobs and Training in the 

1 980s: Vocational PoUvy and the Labor Markets edited by Peter Doeringcr 

and Bruce Vermeulen, Boston, MA: Martinus Nijhoff Publishing. 1981, 
WentHng, Tim. and Tom Lawson* Evaluating Occitpational Edueation and 

Training Programs. Boston, MA: Allyn and Bacon, Inc., 1975. 
Wilms, Wellford W, ViKational Education ami S(K ial MtMity: A Study of 

Public and Proprietary Behind Dn>poHts and Gradimtes. Los Angeles: 

University of California, 1980. 
Zomitsky, Jeffrey et al. Measuring the Costs of JTPA Program Participatiim. 

Cambridge, MA: Abt Associates, Inc.. 1984. 
Zomitsky, Jeffrey et al. Post-Program Performance Measures for Adult Pro- 

grams Funded under Titles UA affd Hi A of JTPA : Early Findings Prepared 

for the U.S. Department of Labor's Performance Starulards Advisory 

Committee. Cambridge. MA: ABT Associates, Inc., 1985. 



O * :'^ 
A- 



4 

Evaluating Program Implementation 

David GaMnbiwski 
Deparmwnr of Health Sen icvs 
VniviTsity of Wushinffion 
with 

Ann Bonar BlaltK:k 
Washington State Emphyment Security 

Whai iiesi rihes dws not i\\f>Um: patterns are not prmrsses hut arc 
thi esuhs tfthem. 

Cons!uncc l^rrin 

Evvrythm\^ in Its Plat e 



Pitxrcss evalualiwis differ from outcome studies in the kinds of ques- 
tions asked, the nature of the data collected^ and the range of methods used 
to gather and anaJy/e information to answer these queslioas. These 
differences arc the basis forgiving n^ore emphasis in this chapter to issues 
that cut across social prc^nims. and for providing a less intensive 
inraiment of these issues in the case example, JTPA. In general, the 
chronology of thischapter follows that of chapten* 2 and 3. Tte first section 
explores a conceptual framework for process evaluations, the second 
examints measurement challenges, the third focuses on methodi^logy , and 
the fourth illustrates scmie of these issues through JTPA. 

The emphasis on gei^ric issues in this chapter fits the nature of stiKlies 
of program implementation. Such studies have a substantially shorter 
history within applied research, the influences and rclationshii^ to be 
studied are not as well-identified or as easily defined, and a miKrh broader 
range of methods can be used. Unlike outcome evaluations, there is no 
established set of tr^itional ccwKeptual frameworks or research strategies 
on which to rely. TTye cvaluaior must develop a framework anldraw from 
a wi(fe spectrum of methods in tailoring a research approach and strategy 
to fit the particular questions guiding an evaluation, within the research 
resources available in a given setting. 

229 
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A Comreptiial Frameivork for Ptocks Evaluations 

Social programs arc created to carry out social policies. More specifi- 
cally, they are to achieve certain outcomes through the use of fmrticular 
interventions^ or change agents (Pressman and Wildavsky 1 979; Shortell 
1%4). TT^ typical reasoning is: "If wc provide X, then Y will result, 
where X is the servicc(s) and Y is the intended outconie(s)/* TTiese "if- 
tten** relationships are a program's *1heory of cause and effect/* This 
ti^ry proix>ses that particular kinds of program interventions will 
produce certain desir^ outcomes. In chapter 1 it was profmsed that two 
kinds of program interventions, or "strategics," are involved in this 
causal theory: an implementation stratef^y and a senice strate}i\\ The 
service strategy is frequently the intervention for which a program is best 
known. However, tte way in which this service intervention is delivered, 
oiganizEtionally, is also an important part of the theory \s explanation of 
cause and effect. 

In tlie past, program evaluations have focused mainly on the important 
reiaiionship between the service intervention and program outcomes. 
Consequently, little empirical knowledge has been produced about the 
influence of the organizational context in which services are provided* 
Yet, those involved in the operation of social programs have long 
recognized that implementation structures arKi methods have significant 
effects on outcomes and are often more open to mc^JificaticMi than is the 
more politically visible service stiBtegy. Fortunately, a new interest in 
evaluating program implementation has cteveloped over the past decade, 
stimulated by practitioner information n^s and interests, and by the 
broadening of the applied research repertoire* 

In this chapter, the success of a social program is assumed to be 
(fepcncknt on both the ai^nx>pnateness of its thewy about the relationship 
between interventions and cmtcomes, and how well this tl^ry works 
when applied in a pragmatic program setting. It also assumes that it is 
possible that a program *s service intervention may be appropriate to the 
problem tl^ progrBm is to aiUrcss, but the implementation strategy may 
be flawed, or that the implementation mode may be a feasible cme, but the 
services provi(fed are not appropriate to tl^ problem. 
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To achieve the outcomes desired, ii is assumed that both the implem- 
entation smitegy and the service smitcgy must be appropriate to the 
problem and operate as intencted. Furthermore, it is understood that 
pn;>blems in implementing a program aiid prc^lcmd in exposing clients 
to tlK? service intervention are directly amenable to change — that is, to 
modification — if sufficient information is available about tlie nature of 
these problems. 

Although few implemeniaiion studies arc designed specifically to test 
a program's cause-effect thecwy, there is a new apprcciatiOT of the 
importance of implementation studies, the complenwntary nature of 
information from process and outcome evaluations, and the utility of 
these different information sources in making policy recommendations 
and program improvements. In fact, tte exploration of how a program's 
implementation and service interventions arc being applied in local 
settings is now considered an integral part of comprehensive evaluations 
of social programs. 

While impact evaluations inform us about the influence of a program's 
service strategy on outcomes, they do not explain why these outcomes 
ocf jrred. Proceiks evaluations fill this kiK)wledge gap by analyzing the 
implementation strategy that contributed to the outcomes observed. A 
primary goal is to answer questioas about how and why programs arc 
working or not working as intencted. 

Tliis chapter presents oi^ approach, among a number of pc^sible 
approaches, for conducting process evaluations. It is based on a simple 
principle: because most wc'ml programs are implemented by organiza- 
tions, imcterstanding what goes on inside and between organizations 
involved in the operation of a program is vital to explaining program 
performance. TTiese organizational relationships are the "^black box** that 
has traditionally been neglected in studying programs. In the approach 
taken here a senrial program is vieweJ as an organizational system 
composed of interrelated j^rts that must Mvrk together to produce the 
outcomes desired. 

In carrying out a process evaluation, the organization is broken down 
into its parts, and each part, as well as the relationship among parts, is 
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e?&amir>ed to learn more implementation generally, or more spe- 
cificaily* the imwns for a particular level of performance. When process 
evaluations are used regularly to evaluate of social programs, they are an 
essential management tool for explmning outcomes and resolving im- 
plementation problems, as well as for improving the general functioning 
of programs. 

Consistent with this approach, a model of an or^anizatv nal system 
and its tnvimmtent is presented as a guide for eva}ua.Mig implementa- 
tion. There arc alternative approaches for studying organizations, but the 
dominant approach is systems analysis, which is the basis for the model 
used terc. TTiis mo<tel (figure 4. 1 ) is a synthesis of a variety of systems 
analysis approaches (Lyden 1975; Mintzbcig 1979, 1983: HoUingsworth 
and Hanneman 1984; Grembowski 1983, 1986, 1989). Tte different 
components of the model arc the •'parts*' of the system. The linkages 
represent the flow of ^sources and relationships among the parts. The 
modeKs perimeter is the larger environment in which the organizational 
system operates. 

Some of the parts of the organizational system, such as authority 
hierarchies, are struvtural, and some, such as actual organizational 
pn^tices in utilizing program resources, are funviional. The underlying 
assumption in the model, and in most system models, is that organiza- 
tibial effectiveness and efficiency depend on the decree of integration, 
or consistency, among the system's parts, and between tte system and its 
external environment. Therefore, the way in which a program's implem- 
entation and service strategies arc actually being applied can be studied 
by analyzing the characteristics of these components and their interrcla- 
tionships, and tl^ rclationship between this organizational system and its 
environment. 

The model may seem to have an air of finality about it, as if it perfectly 
matches what actually occurs in social programs (Mintzberg 1983). This 
impression is not intended. All models arc simplifications of reality, 
which assist the evaluator in managing the study of inherently complex 
phenomena. Conceptual models of this kind sharpen tl^r major featurcs 
of organizations, making tlK^m easier to understaml and analyze. 

This chajMer will give special attention to this framework for process 
evaluations, focusing morc intensively than d » the net impact and gross 
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outetmie chapters on this first theoretkal step in the research pixx-ess. 
This attention is justified by the lack of guidance from previous research 
about what should be studied in analyzing implementation, and the 
multiplicity of conceptual options from which an evaluator must choose. 
TTw general framework for an evaluation provides a context in which 
specific Inarch questions can be developed. These questions then 
direct the selection of a research design and methods of data collection 
and analysis. Therefore, the dexelopnwni of a framev\t)rkfor research is 
the first and most important issue that must he addressed in any prmess 
evaluation. 

Tl» evaluator conducting net and gross outcome studies can draw 
from previously tested and refined frameworits that suggest a fairly 
circumsciibed set of alternative research questions. Relating services to 
outcomes can be very complex, but the nature of the variables and 
ielation.ships to be studied arc often relatively straightforward. This is not 
the case in studying implementation. A wide variety of variables and 
relationships can be the subject of study, and there are few established 
frameworks to guide this kind of analysis. 

The questions to be answered in net impact and differential gross 
impact studies require rigorous methodologies that are more dependent 
than iMTOccss studies on the use of advanced statistical methods. On tl^ 
other hand, because many process variables can only t)e measured 
qualitatively, process studies can draw from a broader continuum of 
established methods of data collection and analysis, such as the case 
study and the social survey. 

Consistent with the .strong emphasis in this chapter on the framework 
for conducting process evaluations, the components, or organizational 
parts, of tte organizational model presented above are briefly described 
in tte sections that follow, 

Tht Environment 

An organizational system can be viewed as having a boundary that 
separates it from the larger environment in which it operates. Everything 
outside this boundary can, for purposes of analysis, be consider^ part of 
tJ» environment of the system. In this model, the organizational system 
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refers to those organizations within a particular service ctelivery area that 
are i«sponsibie for a program. Thcs^ will be refemcd to here ^ program 
organizations. Tt^ system may involve cm central oi^ganization with 
both administrative and service delivery function;, or it may involve m 
administrative organizaticm with a number of subcontractors wt%> deliver 
services, these arrangements being dependent on a program's {Kulicular 
implementation strategy. 

Tl^ environment includes ollKr organizations in the same service 
delivery area which operate ixriaied programs or deliver related services, 
and organizations within and beyond the service area— such as state and 
federal ageiKies — that affect the system through such n^hani^s as 
laws, policies, plans, regulations, or administrative directives* Because 
the environment normally involves influences over which program 
organizations have less than optimal control, it often ncpresents fixed 
conditions that act as constraints on decisions and action, to which the 
organizational system must adapt in order to maintain itself. Information 
about the environment is, therefore, useful in understanding how exter- 
nal circumstances shape program implementation, why certain kinds of 
organizational behavior and piDgram outcomes arc defiled as pmblems, 
and what impact the program may be having on the environment itself. 

Inputs, Key Decisions, and Outputs 

The environment is the soun.^ of inputs that the organiziitional system 
uses to achieve its goals — i.e., the goals of the program, and also its goals 
as a system. Inputs consist of resources, such as funds, staff, and clients, 
and information, such as data on the local economy. 

The flow of input:; into the organizational system is governed by three 
kiiKis of key decisions: revenue, personnel, and program access deci- 
sions. Revenue decisions (ktermine issi^s such as budget allocations for 
achieving goals. Personnel aixl access decisions determine issues such as 
who is hired and fired, and whetter consultants are neected to provide 
expertise in certain areas. Access decisions determine issues such as tl^ 
characlenstics of clients entering the system. The system utilizes, or 
converts, inputs to produce desired outputs^ such as tte number of clients 
whose social or economic problems have been reduced after being 
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expend lo the pnognim. Outputs, in turn, have an ifupavt on the 
cnvironnicnl. cither by ameliorating or exacerbating the pa>blem u 
program is meant to addrCKK. 

Governance, ManMgement, and Feedback 

The conversion of inputs to outputs is directed by j^overnam e-^Ahal 
is, by those individuals with ultimate responsibility for a pn>gram*s 
performance and tlie actions ihey take. OjK»raiionaI responsibility is 
usually delegated by governance to program management. A primary 
function of governance and management is to establish the gcxils of the 
organizational system and make sure they are achieved. Together, the 
output and impact components of the model measure prognun jvrfomi- 
ance. Through the feeJhurk prm exs. program outcomes and their influ- 
ence on the program's environment inform governance about how well 
the sy.siem is performing. If governance receives information indicating 
that the system operating the program is not achieving its goals and 
objectives, the conversi.m process — the utilization of resources — may 
be altered to increase performance. If the impact of the system on its 
enviit>nmeni is pt oblematic. new relationships may need lo be developed 
with OTganizatimis outside the system 

The Conversion Process 

The { onversion prm ess can be studied along four dimensions: mi.%- 
sim, work, t ihtrdmation, and sth Utl climate. In the model, the mission 
dimension consists of the goals and objectives of the organizational 
system, which in many cases an.* developed by governance through a 
formal planninf* proi ess. The miasum amiponent has an important 
influence on the conversion proces.s as the interface between the external 
environment and the other functions involved in the utilization of 
resources. Governance must interpret the system's environment, define 
the system's purpose in this envirtjnment. and design its work, coordina- 
tion activities, and social milieu to accomplish that purpose. Jf gross 
errors are made with respect to defining the mission, these crrt>rs are 
often repeated in designing and monitoring other functions. 
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The working ussuinpliim here is that if the organi/alion is lo cjXMaic 
properly, a "fit'' — or consistency — must be mainiained between the 
environment and the mission of the system, and between the mission and 
the other dimensions of the amversicm pnHressJ An essential part of 
studying the conversion pitxress is determining the extent to which a 
proper fit exists, and identifying what factors aa* enhancing or acting as 
deterrents to maintaining consistency amoiig compinienis. The concept 
of integration or consistency will be discussed in nuw detail later in the 
chapicr. 

The \wrk dimension is an organ i /at icmal system's major means lor 
goal achievement, i.e., the major activities it undertakes \o achieve its 
mission. This dimension includes responsibilities Mich as the prtKHia"- 
mcnt of resources^ the dcveU^pmenl of work How paKcdures, l\\c design 
and maintenance of the pathway clients follow through the system, and 
the provision of services lo thc>^^ clients, TTie work dimension, nepa^senl- 
ing essential organi/ ationa ind service deliver)- functions for achieving 
key goals, is critical lo study in examining the itnplemental iiin of a 
program's distinctive service delivery strategy. 

As suggested previously, the i ooniinaium dimension addresses the 
necessary integration of an organizational sysietn's mission with its 
work effort to produce the intended outcomes. In studying this dimen- 
sion, theevaluator would analy/c. for example, the alkxation of a'spon- 
sibility among various divisions and personnel within the system, 
communication patterns within its authority struciua\ and prmesses for 
developing policies and priKcdua^s. In many swial programs, coondina- 
Mon also involves a number of other factors: iniemal mechanisms for 
ccwnlinating the activities of an administrative agency with those ol 
suba>nlractor organizations pawiding ser% ices; service delivery activi-^ 
tics across subcontractors: and the systeni*s administrative and ser\ ice 
delivery activities with those of impi>rtant organizations in the environ- 
ment. The way in which coordination is handled is considered to have a 
strong effect on the ojH?ration of the system and is, t herefc^rc. a significant 
area for study. 

The last dimension. Mn iui i limau\ refers to the interpersonal internal 
environment of the organizational system, such as the swial worms and 
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profeH«»ipnaI uriemations of staff* staff ami client monile« the motivation 
of staff to achieve the mission, and the level of tension and conflict within 
the system. TTiis dimensicm has a special relationship to the other three 
pans of the conversicMi process: to achieve its gc^ls and survive in its 
environment, the system must define its purix>se, determine what kinds 
of efforts are required to realize it, and apply these strategies with a 
minimum amount of interdivisional and imeipersonal strain (Lyden 
1975). 

Conflicts inevitably cnrcur as resources are utilized. Such conflicts can 
undermine a system 'sefiectiveness and threaten its continuation. There- 
fore, organizational systems operating prc^roms seek to maintain ten- 
sions and conflicts at reasonable levels through various mechanisms, 
such as involving clients and other interest t^oups in planning pnxrr sses, 
or pursuing an "open-door" management style. Inasmuch as the social 
climate dimension plays an imjx)rtani role in accomplishing organiza- 
tional missions, its investigation provides useful information to the 
cvaluator. 

In summary, an analysis framework is proposed that treats the organ- 
izational system operating programs as composed of a number of 
essential pans that aniculaie with one another and with the outside 
environment to affect the way in which programs arc implemented. 
Process evaluations can usefully focus on each of these component parts, 
and on the level of integration and consistency across them. 

In the next sections, measurement and methodological issues in 
applying this, or any other, model in process evaluations are di.scussed. 

Measurement Issue!; in Proce^ Evaluations 

Several factors frequently motivate process evaluations: the desire to 
resolve a specific performance problem, such as increasing the number 
or changing the composition of clients .served: the desire to carry out a 
comprehensive review of a program's operations prior to the begiiming 
of a planning cycle: or the wish to explain the results of an outcome 
evaluation. In some cases, process studies will be limited to those aspects 
of organizations that have received the most political attention. Some 
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eva]ua!ioas may be confined to p^icular components that are known to 
be problematic* Otli^r process studies may emfdiasize compcH^ts that 
are woiicing welU with the purpose of identifying and learning more 
about thc^ features that are mosl critical to replicate elsewhere. The 
more comprehensive procei^ evaluations will look at all components of 
tl^ system and their interrelationships. 

Regardless of the rationale for a>nducting process studies or establish- 
ing their compiehensivei^ss, there is a key difference between process 
and outcome evaluations that has an impact on the measurement of 
implementation influences and on the methodologies used in studying 
them. Two goals of all program evaluations are ( 1 ) to sort out what is most 
responsible for program outcomes and /jtw ttese influences may be 
affecting outcomes, and (2) to make recommendations for conwting, 
sustaining, or improving outcomes by making changes in tl^se critical 
factors. In most programs, despite the simplicity of their descriptions in 
legislation, these influences are enormously complex. Both the implem- 
entation strategy and the service strategy involve an amiy of more 
specific treatments. The relationships among these treatments-within- 
treatments are also complex* 

In conducting outcome studies, cvaluators have been able to use 
previous research to narrow the range of variables studied, with some 
confidence that the most significant factors have been included. Even so, 
there has been a temiency in some outcome studies to assume that this 
carefully selected set of variables, representing particular client charac- 
teristics, services, and outcomes, are the only ones of importance. In 
truth, the effect of a program's implementation strategy cm outcomes can 
never be totally separated from the effect of its service strategy. Implem- 
entation factors are inevitably confounded with service treatments in 
producing program effects. Therefore, comprehensive evaluations look 
jointly at both kinds of interventions. 

The sel^tion of the most important influences in implementing a 
program have not benefit«J from the long research history characteristic 
of outcome evaluations. A host of factoni is involved, and evaluatoi^ 
have not h^ enough experience in studying them to conclude, with the 
same level of confidence, which of these influences tend to be the most 
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important in producing outcomes — cither independently, or in combina- 
ticm with tte service treatments. 

If we are noi able to distill a set of variables for studying implemen- 
tation, then process evaluations must, of necessity, be approached quite 
differently than outcome studies in terms of the questions directing them, 
the measurement of the influences implied in these questions, and the 
methodologies used in answering such queries. Otherwise, tl^ cvaluaior 
will be making premature judgments about tl^ critical influences, and 
may select the v/rong ones to study. 

Given this context, it is still essential in process evaluations to 
formulate a manageable set of geneml research questions, determine 
how feasible it is to answer them in terms of staff, data, and cost, and 
consider the measurement and methodological issues involved. A cx)n- 
ceptual model, such as the organixaiional model suggested here, is a 
useful tool for focusing a process evaluation, highlighting those aspects 
about which there is some knowledge or clue, and tailoring the study to 
the concerns and interests of evaluation sponsors imd users. 

Measurement Approaches 

The choice of variables, or influences, to study in process evaluations 
flows from the research questions being asked. Looking at the compo- 
nents o^the model, it is obvious that even though the research questions 
may be clear-<:ut. the definition of "organizational parts'' and "relation- 
ships" is a very difficult task. 

Although the definition of treatments and outcomes presents problems 
in outc^ome studies also, another difference between process and out- 
come evaluations involves the ease with which the variables selected for 
study can be defined and measured empirically. Again, outcome studies 
can rely on previous research, which typically directs the evaluator to a 
circumscribed set of quantitative indices. Many of these are accessible 
from ongoing administrative data systems~i.e.. treatments, such as 
"classroom training" and "on-the-job training/' and outcomes such as 
jobs obtained, hours worked, wages received These indic^cs can be 
extracted and inserted readily into bivariate (two- variable) and muhi- 
variate (multiple-variable) statistical analyses. In this respect, outcome 
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studies can nionr efficiently simplify the complex aiutionships between 
service interventions and outcomes than process studies can simplify the 
complex relationships among difTercnl aspects of program implementa- 
tion. Although process evaluators may be able to develop measures that 
arc just as accurate representations of implementation vaiiables as 
indices used to define service and outcx)me factors, they do not have ihc 
luxury of using ready-made, easily obtainable indices tliat economize the 
reseaa*h effort. 

Therefore, the firs! challenge in using the mtxlel, and one of ihe 
greatest reseanh hurdles in implementing prtHvss evaluations, is the 
definition and measurement of those characteristics of the components 
of the organizational system that are the major focus of a particular 
evaluation. As with the development of a cnmceptual framework, this 
challenge must also be resolved at ihe front end of the research process. 
TTie research questions based on this framework should clearly direct ite' 
evalualor to tho;^ organizational components and relationships of great- 
est interest. Then the issues are: how will the major variables be defined^ 
how will indices to represent them be developed, what research designs 
are appropriate, and what methcxls can he used in studying an organiza- 
tional system?- 

In general, process evaluations pi>se different measurement and meth- 
odological issues than do gross outcome or net impact evaluations 
because of differences in research purposes and information uses, the 
kinds of data collected, and the range of methods used to analyze these 
data. 

Evaluation Purposes 

Social pmgrams are abstract concepts until applied in actual .settings. 
In the process of translating these abstractions organizationally, the 
iniemkd implementation mode and service strategy are inevitably molded 
by organizational foRCs. It is the purpose of process evaluations to 
examine how well a program's implementation strategy and servic*e 
intervention have been put into practice within the intent of a program's 
authorizing legislation, and to make a judgment of role of implemen- 
tation in pnulucing program outcomes. The information obtained from 
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such an examinalion and judgment provides a basis for dccisioas not only 
about what ne^s to be changed in improving a program, but /frw 
changes might best be made. 

As Patton (1987) points out. however, choosing questions to answer 
in process studies requires the acceptance of a tradeoff between breadth 
and depth. A few questions may be studied in great depth or many in less 
ctetail. The former may provide clear results on specific issues but fail to 
address other important concerns. On the other hand, collecting less data 
on a wider range of issues often reduces confidence in the findings. 
Breadth is often sacrificed for depth when the goal is to explain an 
outcome or gain insight into the cause of a problem. Identifying a few 
important questions to study may be critical if it is felt that these issues 
affect other characteristics of implementation. 

The purpose of outcome evaluations, in contrast, is to determine the 
specific relationship between the service strategy and outcomes — ide- 
ally, whether the service intervention has been responsible for the 
outccMnes or these outcomes have been due to some other set of influ- 
ences or to chance. The information from outcome studies informs 
policymakers and administrators about the effectiveness of the service 
intervention, which is critical to decisions about who should be served in 
the future, and with what mix and sequence of services. This purpose and 
me is essential, but narrower in scope tlian the focus of most process 
evaluations. 

Qualitative vs. Quantitative Measures 

The task of developing operational definitions and indices for key 
variables in studying the components of a program's organizational 
system is also much more difficult in process evaluations because a great 
(fcal of the data on implementation are qualitative in nature. ' Dependmg 
on the purpose of the evaluation, the attitudes and behavior of significant 
participants within and outside the system may need to be defined and 
measured. Ways to measure the content of decisions, the characteristics 
of decisionmaking entities and processes, and the nature of service 
delivery policies and practices al.so may be necessary. The personal 
attributes and actions of clients, management, and other actors may need 
to be defined and classified. 
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Fot example, ••profiles" of individuals and services may have to be 
(kveloped using a combination of indices of difTerent variables, such as 
profiles of the different kinds of clients entering the service ctelivery 
system, those provided services, and the combinations of .^rvices they 
received. Defining tl^ ^hard-to-«:rve" client population, for instance, 
may r^iuire the development of a set of weighted social and economic 
iiKiicators that can be quantified. 

However, quantifying variables that are nomially described by using 
qualitative measures, such as the hard-to-serve, require the evaluator to 
make arbitrary decisions abwit which indices are the most valid and 
useful, how highly correlated are these indict^s with one another, and 
which should be given more weight than others in nepresenting tte 
phenomenon being studied? Many implementation factors, such as 
service delivery practices, cannot or should not be quantified. The 
tradeoff is that qualitative infomiation is necessarily classified and 
analyzed more subjectively, with less reliance on the use of the statistical 
methods that characterize many outcome studies/ 

Therefore, implementation evaluations cannot depend as much as 
outcome studies on existing management information systems (MISs), 
Tte measures in these automated infomiation systems tend to be simple 
quantitative data elements that satisfy reporting i^eds. They are often 
restricted to a small number of variables and arc frequently limited to 
single indices for these variables. But an MIS can be an important 
resource for outcome evaluators. For certain aspects of implementation, 
cmgoing monitoring/reporting systems can be of some assistance to 
process evaluators as well Certain aspects of implementation arc quan- 
tifiable — particulariy the characteristics of the client flow through the 
service delivery pathway. Some characteristics of the client pathway, 
such as the number and kinds of clients enrolled, assess^, and assigned 
services, are ali^ady being collected in these information systems. 
Additional questions cm intake and follow-up forms can capture some of 
what is currently absent from these MIS systems. 

Statistical analyses of these data can then be carried out to determine 
some of the details of the convention process (see figure 4 1 ). The chapter 
cm gross outcome evaluations covers this aspect of process studies, and 
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will be discussed in more detail later in this chapter. However, despite the 
ability to use quantitative measures for :;ti'dying certain elements of 
implementation, the fact that many of the variables studied in process 
evaluations cannot h: measured quantitatively means that the evaluator 
must use innovative measurement approaches and make a careful selec- 
tion of a combination of methods for imalyzing this information (Patton 
1987; Shorten 19X4). 

In summar>*. there arc several critical measurement issues in studying 
implementation: (1) the extent to which useful, valid, and reliable 
quantitative data elements arc available for defining important variables: 
(2) the extent to which a program's ongoing MIS contains reliable, 
extractable. and accurate data on these variables; and (3) the extent to 
which useful qualitative definitions and indices of variables can be 
develoiK'd and col leaned fi)r factors tha! do not lend themselves to 
quantification. 

Methodological Issues in Proces; Evaluations 

Performing a process evaluation within the organizational approach 
proposed is much like assembling a jigsaw puzzle. After the evaluation's 
questions are identified, definitions and measures of the im{X)dant 
factors are developed, a research design is selected to guide sampling, 
data collection, and data analysis, and information is collected about 
each comjxjncnt of the system and its environment, the process evaluator 
must infc'iiraw this information to provide insights about interrelation- 
ships within the organizatitjnal system and tlw system's relationship to 
itsenvin^nmeni. These insights will help the evaluator make a judgment 
of the efficiency and effectiveness of program implementation, and its 
probable infiuence on outcomes. 

The integration of this kind of information by a researcher or research 
team is a combination of art and science, since precise statistical 
estimates canmrt be made about most of these relationships. It is 
imfx>nani. given this constraint, that the researcher select the most 
rigorous melhtxiology pt>ssible. in order to make sound judgments based 
on the information available, and to have a credible basis for defending 
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the!5e judgments. Therefore, the xeUrtion of a research tlesiffn is the 
second cihiUenxe in planning studies of proffram implementation. 

TTic purpose and 5icojK of the evaluation* the nature of the leseaah 
questions, and what is already known about the key variables and their 
rclationships efTectively shape the choice of a research design that will 
guide the sampling of subjects, data collecticm methods, and methods of 
information analysis. Other important considerations aiv the kinds of 
data available, their quality and accessibility, and tlK?ir appropriateness 
for statistical analysis. Whether information to answer the research 
questions must be freshly collecled, such as through questionnaires or 
interviews, or is available from existing administrative data systems is 
also a determinant of design decisions. 

The Research Design 

A prerequisite for deciding a hat rcseanch design is most appropriate 
and feasible is a clear understanding of the lesearch questions to be 
answered. This includes decisions about what factors are to be studied 
and what relationships between them are to be analyzed. In chapter 1 , 
research designs that guide evaluation activities in answering the re- 
search questions are classified into four general categories: exploratory, 
descriptive, quasi-experimenial. and experimental. Hie guide for re- 
search that most closely approximates scientific principles is experimen- 
tal design, which involves the andom assignment of research subj«rts 
into ^'program-treated" and "no**program-treated** groups. The outcomes 
of these two groups arc compared stalislicalls to determine whether the 
program's service intervention is producing intended results. 

Experimental designs are appropriate, however, only when there is 
substantial knowledge of the key independent and dependent variables, 
which can be measured quantitatively with little em>r. C^herwise these 
designs may be inefficient and may direct research attention to the wrong 
variables. These designs are the preferred choice for net impact studies 
in fields such as employment and training. The U.S. Department of 
Labor's current national field study, the J TPA Experiment, is an example 
^Bangsor et al., 1988). In some programs, hiiwever. such as certain 
mental health programs, ihe literature may be equally extensive but some 
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of ihc critical outcome^^ — increased sclf-cstccm, more realistic life 
expectatims, and improved quality of life — arc very difficuU lo measure 
and quantify. 

Experimental def;igm are not always appropriate for studying the 
effects of service interventions on outcomes in such programs. In some 
cases the evalualor may be forced to develop quantitative •^proxies'' for 
mental states and behavior that have qi^tionable accuracy. In other 
programs, new interventions may be tested for the first time and lack a 
substantial history of evaluation. Experimental designs are not appropri- 
ate where the major variables of interest are yet to be identified or clearly 
cfefined. 

With many social prog)'ams« quasi-experimental desigiLs employing a 
comparison group are a useful fall-hack when the political or organiza- 
tional context of a net impact evaluation preludes withholding a service 
treatment from one group of eligible clients. These designs are no more 
appropriate than cxj^rimenlal approaches, however, if cither the serv- 
ices or the outcomes are difficult lo measure and cannot be incorporate 
easily into statistical analyses. 

In practice, process evaluations primarily utilize exploratory and 
descriptive designs. These designs reflect a wide range of research 
approaches, from very simple case studies to very technical social sur- 
veys. Any given process evaluation is likely to involve a customized 
ctesign that combines more than one type of appn^h within these two 
general types of designs, bas^ on the questions directing the study and 
what organizational comporants and relationships arc of greatest interest. 

For example* customizing an evaluation of the implementation of a 
basic educational skills program might involve an exploratory design for 
studying program decisionm^ing. a simple descriptive design for 
stiKiying ite nature of progiam participation* and a more sophisticated 
design involving statistical analysis of client flows. Studying govern- 
ance and management, for instaiKre, involves a range of organizational 
behavioi^— negotiating, decisionmaking, selecting rules and regula- 
liws, and monitoring compliance. It also involves stoiciures that permit 
and shape this behavior — official policymaking bodies, power struc- 
tuies, policy statements* mmiuals, and reporting requiren:enls. To study 
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this component of the modeI« tl^ evaluator may initially want to identify 
key factors by conducting in-depth inlcrviewji of governance and man- 
agement. Hits may be followed by a moit^ Kpecific and detail^ denciip- 
tive survey of program ckrcisionmakers, i.e., those whose decisions hpve 
an effect at the client leveK and clients themselves. 

An innovative, customized approach is essential in conducting a 
comprehensive process evaluation that fixruses on all the com{X)nentH of 
the organizational model and key relaticmships among them. This may 
involve an integration of i^veral different exploratory or dei^riptive 
designs focusing on diffeit^nt aspects of the otganizational system to be 
studied. The information yielded by this customized appioach miLst then 
be utilized by evaluator, l^sed on clearly deftn^ criteria, in judging 
the ^fit'* achieved among tt^ different parts of the orgmiizational system 
and the relationship between the program organization and its environ- 
ment. This requires a great deal of research and program experience, 
creativity, and a consideration of all the methodological options. 

In conclusion, exploratory process evaluations are important prede- 
cessors of outcome studies, for they can identify influences in the 
organizational system that af fect progi^ outcomes. Exploratory de- 
signs are also useful in identifying key implementation variabiles for 
inclusion in subsequent impact evaluations. Hie gcKil of these explora- 
tory studies is to reflne the research questions that could subsequently be 
studied, and to pin down If^ most important influciKes to study, using 
more rigorous designs that rely on quantitative methods. 

Sampling 

In studying program implementation, we are not only interested in 
individual program participants, who are usually the focus of most 
outcome evaluations. The research subjects in process evaluations may 
be program staff, such as directon?, planters, managers, and casework- 
ers« or individuals outside the program's organizational system who 
interact with p^icipants or the system but are not formal targets of tlK 
program's interventions, such as participants* families, other service 
IMt>viders, employees, or school personnel Or the '^subjects of study** 
may be far less tangible aspects of the system or its environment, such as 
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the components and processes of the organi/aiiun. Therefore, the ques- 
tion — who OT wlwt is to be sampled — requires a more complex answer 
in process evaiualions than in outcome studies. Most pn^ess evaluations 
seek information on a vomhination of human subjects and a ran^c of 
oi^ani/aiional phenomena. 

When individuals arc the subject of study, it is often mon: economical 
in terms of time, etTort. and money to study a small group of individuals 
who represent the lai^er population of interest, such as a sample of 
personnel who arc representative of a program's entiir case management 
staff, or a sample of program participants who are representative of a 
particular subpopulation within the target group. The way in which the 
evaluator selects these "representatives," and how many arc selected, is 
critical in preventing bias and in obtaining accurate estimates, as dis- 
cussed in chapter 1 . If the prcxress evaluator wants to draw a sample of 
those individuals who best repnrscnl a larger group, basic sampling 
principles are necessary to reduce bias and allow for the generalization 
of results to the same program in a different setting, or to other similar 
programs (Kish 1%5). 

Prohiihitity sampihifi is an efficient method to use in studying client 
flow alcHig the service delivery pathway, mainly because of the large 
number of clients involved and the ability to list all individuals in the 
population of intercst. This form of sampling assures the evaluator that 
results are representative of the larger population. 

Where the general characteristics of the entire caseload are the main 
interest, simple random saniplhi\( is useful. This allows each case in a 
population— and all combinations of these elements — an equal chance 
of being included in the sample. 

More often, tte evaluator will want information broken down by 
(temographic, labor force, or other characteristics. In this case, a sirati- 
fied random sample is appropriate. Here the population is divided into 
various categories, or "strata,'* and a simple random sample is drawn 
from each stratum. These subsamples are then joined to form tl^ total 
sample. For example, in studying the characteristics and patterns of 
client flow through a service delivery system, the prcxress evaluator may 
want to know the experiences of different client groups. Each group 
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serves as a straium, and a se}xiratc random sample of cliems is drawn 
from each stratum. 

Probabiliiy sampling, however, is not appropriate for gathering much 
of ihc information needed in pnxress evaluations. Studies of program 
implementation may f(H:us on oniy a small number of individuals or 
c^imizations: for instance, tho^ policymakers or organizations that 
appear to have the broadest effect on a program. In this situatitm, a wide 
variety of nonprobability sampling techniques will be more appropriate 
and useful Purposive sampling is often the most feasible choice (Paiton 
1987). This involves a careful selection of each individual or entity to be 
included in the study sample using a set of criteria hai^ on the purposes 
of the evaluation and the questions to be answeiwi by it. This is a more 
subjective strategy but is often better suited to the information needs of 
process evaluations. For example, selecting a **panel of experts'* who the 
evaluator has reason to believe is representative of individuals knowl- 
edgeable about a given phenomenon can be more economical, conven- 
ient, and useful than questioning a larger group selected through random 
sampling/ 

The problem with nonprobability sampling, however, is that the 
evaluator has little basis for estimating how representative the informa- 
tion obtained really is,** Interpreting the results of evaluations confitwd 
to this kind of sampling must therefore be properly qualifi^. In practice, 
proc^^ss evaluations usually involve a combination of ite two basic kinds 
of sampling methods. Integrating information obtained by these differ- 
ent meaas requires inductive thinking and often produces insights not 
otherwise available, A wealth of excellent beginning texts is available on 
sampling issues, siralcgies, and specific techniques. Some of these are 
listed in the reference section following this chapler. 

Data Collection and Analysis 

As discussed earlier, studies of implementation tend to utili/e an 
exploratory or descriptive design, and more often, a combination of these 
approaches. The purpose of exploratory designs is to gather beginning 
ideas about a particular phenomenon, identify its most important ele- 
ments and interrelationships, and fomiulate more precise questions for 
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further study. Although some of the characteristics of program implem- 
entation have been studied fairly extensively, such as management and 
service ttelivcry, the more distinctive features of a imjgram -i.e., those 
characteristics intended to be novel, such as the use of performance 
standards for monitoring program outcomes in JTPA — may not have 
been well-studied. These new features arc expected to fiave a greater 
effect than previous program strategies and. therefore, arc an important 
area of inquiry. Exploratory approaches arc most u.scfiil in examining 
these unique program characteristics. 

An exploratory design calls for relatively unstructured methods for 
collecting data and usually nonstatistical methods ior mialyzing iJiis 
information, r^ta collection methods such as participant-ot^rvation. 
oral history tcchniquCvS. ami pancl-of-expens sttT.iegies, which resist 
reliiuice on prior assumptions and preconceived frameworks, are ex- 
amples of standard data-gathering .strategies appropriate to exploratory 
studies. The information collected in this way is processed, distilled, and 
integrated by the evaluaior within the questions of interest. 

Where knowledge of a phenomenon already exi.sts, a continuum of 
descriptive designs can be u.sed in aasw^ring process questions. In 
general, the purpose of descriptive studies is to determine assm iatUms 
among important factore — how these influences are related— y^hx^ 
goes beyond the intent of exploratory studies, but stops short of deter- 
mining caase-effect. Relationships may be assessed using siati.stical or 
nonstatistical evidence. Which source of evidence is the more "truthful" 
depends on the quality of rescanrh design: whether the right variables 
were studied: whether these factors were appropriately defined and 
measured (either quantitatively or qualitatively); whether these meas- 
ures were collected reliably and in accordance with the sampling strat- 
egy; and whether the resuhmg data were appropriately anaiyzoi. 

The least sophisticated descriptive studies can be illustrated by rela- 
tively informal surveys of selected participants, staff, or relevant others, 
using flexible inteniew schedules, or the use of a small .set of general 
interview questions chosen to gather in-depth information related to the 
overall evaluation questions. Collecting this kind of "open-ended" 
information requires a flexible classification scheme for analyzing data 
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and. ultimately, a great of deal of study and integration of information by 
llK evalualor. The advantage of data collection and analysis flexibility is 
that the cvaluator can make immediate adjustments in both Hk structure 
of an interview, based on how an interview is unfolding, and in ^he way 
infomiation from interviews is coded or classified, based on the nature 
of the data emerging ftDm the interviewing process. 

The most sophisticated descriptive designs, which permit advanced 
statistical analysis, require much more stmcturcd forms of data collec- 
tjon — for example, standardized questions with quantitative response 
categories. A number of different survey techniques are possible. Ex- 
amples are one-time surveys that measure a study sample at only one 
point in time, and panel surveys—often referred to as longitudinal 
surveys — that measure the same group of individuals or organi/atioas at 
various points over an extended period of time. Surveys c^ differ 
substantially in terms of the number of people studied, the sophistication 
of the information-gathering mechanism, the complexity of the informa- 
tion obtained, and the kinds of analysis techniques that arc appropriate. 

The automated management information systems in social programs 
used to inform governance, management, and fundcrs about the organ- 
izational system arc es.sentially surveys based on intake questionnaires 
filled out by clients and update forms vompleted by staff. In client 
surveys, infomiation is typically gathered on each client at standardized 
decision points along the service delivery pathway. A new feature of 
many programs is a one-time, follow-up panel survey of clients (and 
sometimes others, such as family member.% or employers) utilizing 
questicmnaires or telephone interviews. These standard administrative 
data collection teclmiques typically prtxiuce a narrow range of informa- 
tion. Their advantage, however, is that the data arc quantified and can be 
analyzed using statistical techniques. 

It is clear, however, that the data collection and analysis choices that 
must be made by t!ie cvaluator are more complicated in process studies. 
The options arc greater, there arc different sets of risks and benefits (in 
term.4 of bias and utility) involved in each decision, and it is difficult to 
identify all the important variabies at the outset; i.e., many must be 
di.scovered as a study progres.scs. For example, if the researcher wants to 
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study ihc conicm of and motivation for particular policy diaxtivcs 
affecting implementation, administrative records are an important source 
of information. Some of this information can even be quantified, such as 
how many decisicmmakers ttwk a particular position. Imw many policies 
were issued and on what subjects, and how many of these were actually 
implemented. 

In many cases, however, information fmm a-cords must be extracted 
laboriously, using only a general framework for identifying the relevant 
influences. Thcevalualorcan supplement this information by interview- 
ing program policynu^kers. or gathering infomiation from decision- 
makers outside the system who are likely W have a special knowledge of 
orogram policies and policymakers, using open-ended or more struc- 
tured techniques. Most typically, the cvaluator will chwse a mix of data 
collection andanahm straivpvs. The nature of the mix depends on the 
kinds of infomiation available, the kinds of evaluation resources that 
exist forgathering and analyzing this information, and the quality of data 
in automated program information systems (Palton 19S7; Mintzberg 
1983b ). 

Statistical analysis lechniqucs arc powerful tools in the researcher's 
mix of strategies. Therefore, opptmunilics lo obtain high quality quan- 
titative data must be exploited. Statistical techniques can provide the 
lollowing kinds of information; (I) information about central tenden- 
cies, such as what staff altitudes may be most typical; (2) infomiation 
abt>ut variations, such as whether clients arc similar or different regard- 
ing .self-selection into a pmgram or some other characteristic: (3) 
information to compare central tendencies and variations across groups, 
such as how the rate of self-selection may vary between white and 
minority clients; and (4) information about relationships, such as whether 
.selection into a program may be related to ( but not necessarily caused by) 
staff attitudes and behavior. 

As in outcome studies, it must be . ept in mind that statistically 
significant relationships are .statements of probability, not certainly. 
Making causal infer'nces requires evidence beyond establishing the 
existence of a relationship, or an association. In nonexperimental studies, 
such as most process evaluations, the evidence is not sufficient to claim 
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causalily, even though findings may be statistically significant. Since 
process studies typically involve a combination of statistical and nonsla- 
tistica! techniques, tlw evaluator must be caieful to judge the importance 
of statistically significant results in the cx)ntext of the results obtaini^ by 
otl^r means, such as through content analysis or some other form of 
classifying information for analysis. 

In summary, a wide variety of data collection c^d analysis methods 
can be useful in pmcess evaluations, since the issues studied arc sufc^tan- 
tially difTerent and considerably broader thsm in most outcome studies. 
The process evaluator must be familiar with the full research i>cpcrlotrc, 
selecting a combination of methods that best fit the nature of the 
questions being asked and the kinds of information dcsiied from a 
particular evaluation. Because there ane many moie methodological 
choices available ihim in outcome studies, the researcher must take an 
inventive appri>ach in customizing sampling, data collection, and analy- 
sis to the sf^ific purpose of an evaluation. TTie interpretation of findings 
yielded by this customized appioach will usually necessitate a unique 
integration of information by the evaluator within an overall analysis 
framework. Fortunately^ piXKress evaiuaiors now have an abundance of 
methodological material in the research literature to aid ttem in this task 
(sec references for selected sources). 

Application of the Organizational Model 
to the Case Example: JTPA 

It is now important to illustrate how a proces.^ evaluation can be 
conducted by applying the organizational model to im existing program. 
This illustration will focus exclusively on Title II of the fcileral job 
training legislation, which is JTPA*s major employment and training 
program for adults and youth. Tlie Title II program is implemented in 
local service delivery areas (SDAs). Governance occurs through a 
Private Industry Council (PIC), composed of members from the public 
and private sectors but dominated by the latter sector as mandated by 
Crmgress. 

The administrative agency of an SDA, which is responsible for 
operating the JTPA program, frequently decentralizes service delivery to 



254 Evaluating Social ProWemjj 



otter organizations in the community through perlbmiance-based or 
other kinds of contracts- Sometimes PICs and administrative entities arc 
one and the same. Where a PIC and an administrative entity have separate 
fuiKtions* the PIC is primely responsible for local employment ami 
training policy, coordination* and program oversight^ and the adminis- 
trative c^anization has operational responsibilities. Mayors arc ex- 
pected to work with PICs as part of a puWic/pn vate partnership. 

Typically, the PlC/local-eic^ted-olficial parti^rship, the administra- 
tive agency, and the contracting private and nonprofit community 
agencies that may deliver services comprise ihc Icxral organizational 
system of greatest interest in process evaluations. TTiese are the organi- 
zations ultimately responsible for program implementation and perform- 
ance. The main features of JTPA implementation are described in the 
appendix. To add to the reality and increase the utility of the application 
of tiK orgmnEjtf ional model, insights and examples from actual studies 
of program implemeniation will be woven into this application. (Comp- 
Ui)Iler General 1985; Cook el al., 1984a, 1984b; Walker 1984, 1985.) 

Each component, or group of components of the model, will be 
discussed separately, in the same chronological order as they were 
introduced in the first section. Among the many potential influences, 
measures, and methods that deserve ccMisideration, only a selected few 
can be covered under each set of components. Three kinds of issues will 
be addressed under each part of the model: concepiuaL measurement, 
and merhodoiogical. Because of its importance, the greatest attention is 
given to the conversion process. 

Studying the Environment 

Conceptual issues 

The environment of an SDA includes local conditions, such as the 
local economy, as well as other JTPA agencies at the local, state, and 
federal levels. These agencies form a complex web of organizational 
relationships that often influence — ^and, at times, dictate — Ik>w the 
SDA's programs are implemented.^ Specific asix:cts of an SDA*s 
environment that should be included in a process evaluation are re- 
viewed briefly l^low. 
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The Immediaie Fmironment of Programs 

The chief task of the evaJuator in studying influences emerging from 
the environment is to identify the factors that operate cloj^si to the 
program and, therefore, have ti^ greatest effect on its resources TTiis 
level of the environment is the city, town, or rural area surrounding the 
organizations implementing the prognun. TTie most important commu- 
nity influences are social, economic, and political, such as later market 
characteristics, market trends, <kmographic characteristics, incideiKe of 
social problems, and attributes of Iwal government* 

For example, information on the area's poverty population, labor 
force, and wage strucmre provides a context for understanding the 
pr^ical limits placed on program implementation, how program or- 
ganizations accommodate local corditions and circumstances, and the 
significance of program outcomes. If, for instance, wages are generally 
depressed for women, or hi^-paying ji>bs scarce, it may be unrealistic 
to expect that changes in program implementation wii! improve female 
participants* situations. 

The Inter-SDA Environment 

Local organizational systems aro influenced also by rciationships 
among SDAs, which may range from a high level of coordination and 
cooperation to severe tension and conflict. Good relationships among 
local-level organizational systems may support greater independence for 
SDAs vis-a-vis \\k implementation of state policies. Disruptive compe- 
tition for state incentive money, for example, can undermine collabora- 
tive action to pursue local-level goals. Theiti'forc, process evaluations 
should explore the nature of this particular set of relationships and its 
potential impact on program operations. 

The State Environment 

Studying .state-level oi^anizations as a source of environmental influ- 
ences should again involve an investigation of social, economic, and 
political characteristics. A major emphasis, however, iJiould be on the 
network of slate organizations that ctevelop stale pr.^gram jK>licics, 
assume slate-level administrative responsibilities, craft state prc^ram 
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plans, formulate criteria for moniloring coordination among state agen- 
cies and inKtilutions relevant to employment and training pmgrams^ 
make s^latewide assessments of UKal performance, and sometimes evalu- 
ate SDA program implementation and outcomes. 

The Federal Environment 

As with state-level organizations, the study of federal environmental 
influences involves questions such as the following: 

1 . What social, economic, and political influences are operating nation- 
ally to affect federal employment and training policy and practice? 

2. What are the specific purposes, policies, rules and regulations, ad- 
ministrative directives, and the actual organizational pi^ctices of 
relevant agencies at this level, such as, the federal agencies and the 
Congress, which have an impact on SDA implementation? 

Altho' ' ^ evaluations of IcKal-level program implementation can- 
not a! jevo!e much time to federal-level influences, it is important 
to ackn^.A'jedge the larger framework within which implementation 
occurs, 

bueracuon Between the Or^anizatUmai System and Its Environment 
The model assumes that program outcomes and what happens in 
implementing programs are tiot totally at the mercy of the environment. 
The characteristics of local implementation have an impact on the 
envinmment as well. While this reciprtKal relationship is difficult to 
study, it is a significant aspect of comprehensive prcK^ss evaluations- 

Measurement and Methodological Issues 

Measurement Issues 

Two kinds of environmental influences, among a wide range of 
potential variables, are imporiatit to define and measurc: ( I ) socioeco- 
nomic and {K>litical factors, and (2) structural and functional character- 
istics of state and federal organizations. Table 4. 1 piovides examples of 
environmental indicators for these dimensions. 

Demographic infomfiation on the area's poverty population and labor 
force, and labor market information on tl^ wage structure of local jobs 
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T8b)e4.1 

Emmptts la Envbtmmsi^i Imlieatois 



1. 08mcHinij»&te 8^ HWit9\ huttcfbra 



A Local economy 
Unemployment rate 
Per capita income 
Average wage rate 
Inflatiwi rate 
Unemployment 

Insurance caseloads 
Numt>er unfiiled jot)s 

B Market trends 
building permits 
new businesses 
business closings 



A Oemogra4)hjcs 
Totfid peculation 
Population by 
♦age 

• rmre/ethnic group 
♦sex 

• education 

• religion 

B Sociai problems 
Cnme rale 

Alarfioiism/drug abu^ rate 
School 6topou\ rate 
Housebokfs wttti female 
h^d (%) 

Households receiving 
stafe/tedemi 
assistance (%) 



Psiitisat 

Pnorrty given 
to emptoymenl 
training 
government 



B Relationship 
between 
gos^rnn^nt 
and biminess^ 
service 
agenc^. 
and client 
advocacy 
groups 



n. Buraaucratie Ctiaracttftette of /Htefsies CouinHis 



Clisradsflslfc 

Decfsicwimalung hierarchy 

Poiicymaking process 
Exercise ot regulatory power 
Adn)inistrat}ve procedures 

Coordinahon with other agencies 
Monitoring 



Indieator^ree of iBftm^si 

Organization chart showing 
accountat^lrty patterr^ 

Poltcy statements 

Wntten regulations 

Service plans 
Administrative diJWtives 

Cooper atiw agreements 

Wntten protocols 
Sanctions levtoJ 



provide a context forjudging the meaning of program outcomes, and the 
constraints placed on program implementation in resolving outcome 
problems. Many program organizations regularly document the kinds of 
conditions suggested as part of their planning process. Normally, indica- 
tors of political and bureaucratic conditions are qualitative^ designed to 
capture influences exerted by political and organizational forces m 
program implementation. While the lists in table 4 J are incomplete, tte 
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intent is to construct a set of indicators of the relevant conditions 
affecting the local organizational system, as a useful measuiemeni tool* 
This helps the evaluator understand, for example, why a wage differen- 
tial bciwcCT men aiKl women may have occumw!. and why this outcome 
may be considered problem^ic. 

While many of the variables in table 4,1 can be measured quantita- 
tively (c.g,, local wages and unemployment rate), others can only be 
measun^ qualitatively (e.g., a written history of state pi>licies regarding 
a local problem). Although numerous indicators can be i instructed, the 
evalualor's choice is usually ctetermined by the purpose of the evalu- 
ation. For example, a process evaluation examining low placement 
wages for female participants might fcxrus on local wage rates, while an 
evaluation of job satisfaction among staff might focus on relationships 
between the stale and the local administrative agency/ 

Methixiolo^ival Issues 

Fecknil and stale employn^nt and training agencies regularly collect 
data on some of Jie variables in table 4. 1 through tl^ir ongoing automated 
management information systems. Tt^ Bureau of Labor Statistics and ttur 
U.S* Census collect acklitional data on a regular txisis. A number of 
national IcmgitiKiinal surveys pwrovide environn^ntal data. Governors* 
offices arei stale employment and welfare agencies analyze state-level 
socioeconomic and labor market infwmalion* Some slates ccxiduct peri- 
odic social surveys providing a range of infcmnaticm on employment, 
training, and related programs, and their environments. TTiesc quantitative 
data are relatively ^cessible for sampling and statistical analysis pur- 
pcxses. Multivariate statii^ical analy^ can be performed to (tetermine 
which variables are most important, and how tl^y nmy be associated. 

Obtaining and analyzing information on political influences, and on 
the characteristics of federal and state organizations i more difficult 
tasks becau^ of the qualitative nature of the data. S ing organiza- 
tional structures and processes often requires an analysis of administra- 
tive recoros within a framework ctevised by ihc evaluator, supplemented 
by informal or structured interviews with key actors to provide greater 
information dejrth and accuracy. 
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Studying how a progimi's organizational system has, in turn, influ- 
enced its cnvinonmem will involve some of the same data sources. For 
example, the cvalualor may want to identify SDA resistance to state 
incentive policies and how this resistaiKre may lead to modifications in 
performance standards. Or, the researcher may want to study the influence 
of exemplary SDA performance in increasing support for testing alter- 
native solutions to problems such as the male/female wage differential. 

In summary, much can ]x leaned unobtmsively about the cnviiDn- 
mem from information already available to the evaluaior from automated 
systems and records, some of which lends itself to statistical analysis. 
Otiwr kinds of data will have to be freshly collected within sf^ific 
information purposes, using purposive .wnples, employing question- 
naire or interview formats designed with attention to analysis options, 
and building in ways to reduce bias. 

Goverance and Management 

Conceptual Issues 

The conversion of resources into the human, organizational, and fiscal 
''products'' of a program is directed by }{(pvernam'e. Governance refers to 
the body of decisionmakers who develop and communicate the goals and 
means to be used by the oi^anizationaJ system in achieving its purposes, 
the rights and responsibilities that guide the behavior of these decision- 
makers, and their actual attitudes and behavior in carrying out tteir roles. 

Management refers to the hierarchies of personnel, under the guid- 
ance and supervision of governance, who are responsible f^^** 'he day-to- 
day operation of organizations within the system, t ities and 
privileges, and their actual attitudes and behavior. The p* .^cements 
and actions of these major program actors influence u, ^rm and 
substance of program implementation. In JTPA, governance is the 
responsibility of the SDA*s Private Industry Council (PIC), the SDA's 
top elected official, and its administrative organization. This arrange- 
ment is similar to the state-level govemarK^ and management structure 
headed by governor. State Job Training Coordinating Council and 
JTPA administrative agency. 

TTirough a feedback process referred to in the model as the perform- 
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ance control system, program outputs inform governance and manage- 
ment about the efiiciency and effectiveness of the organi/itiional system. 
Hie control system will be discussed later, after looking more closely at 
the conversion pnxi^ss for translating organizational resources into 
program outcomes. 

In JTPA, subcontracting for administrative or client services, and 
monitoring subcontractors* performance are special functions of gov- 
ernance and management, Ttere is an emphasis in JTPA on "pcrfomi- 
ancc-bascd contracting/" which commits governance and management 
to regular assessments of subcontractor performance, and leads subciw- 
traclors to engage in self-monitoring activities. 

Memwremtnt and Methodological Issues 
Measurement Issues 

Two kinds of variables are particularly important to examine in study- 
ing governance and management: ( 1) the characteristics of the power 
hierarchies directing the system, and (2) the kinds of decisions made at 
different levels of this hierarchy. The evaJuator must distinguish between 
the kind of hierarchy intended in a program's implementation strategy 
and the one actually utilised, and the areas of decisionmaking expected 
to be given highest priority and those actually given the most attention 
by decisionmakers, 

A distinguishing feature of JTPA implementation is the use of a 
public/private governance partnership. Therefore, an important implem- 
entation question is **How well is this partnership betu een local elected 
officials and the PIC actually working?" Here, key areas of investigation 
are the level of cooperation and collaboration, and the merging of 
expertise and other resources to create a program responsive to tte 
private sector job miricet. The composition of the parti^rship and 
control of decisionmaking are authorized in the legislation. More diffi- 
cult to ikftm are the partnership's intended functions and actual activi- 
ties. 

The JTPA legislation cotnniits governance to a number of genei^l 
responsibiiities, such as setting job training |K)iicy, reviewing job train- 
ing pians« overseeing performance and monitoring coordination. The 
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criteria for formulating policies and regulations to canry out these 
responsibilities, and actual decisionmaking behavior, must be specifi- 
cally defined. The literature on organizational theory and behavior is a 
useful resource for measurcs, but the evaluator will frequently need to 
develop his or her own indices and criteria based on the data already 
available and the data that is feasible to generate in a given program 
setting (Blauand Schoenhcrr 1^7 i: Simon 1957. 1977: Mintzberg 1979, 
1983K 

MciluHiola^h al Issues 

Studying governance involves an exploratory or descriptive approach. 
A traditional data collection device for understanding who is most 
influential in making particular kinds of decisions in given arcas of 
activity, and through what means, is the sele-^tive interviewing of known 
decisionmakers, using largely open-encted questions. A content analysis 
of this information is useful in sketching the nature of decisionmaking. 
On the basis of this preliminary information, more structured interviews 
can be teld with key decisionmakers, and a more thorough study of their 
documented positions can he made. In analyzing such information, the 
evaluator will want to look for discrepancies between the intended 
governance and management strategy and actual practices, and the 
influence such discrepancies may have on other components of the 
system — particularly service delivery and outcomes. How^^ver, siiKe 
most of the data on this comiK)nent will be qualitative, statistical analysis 
techniques will not be feasible, and the evaluator will need to develop a 
scheme for classifying data that fits the questions to \k answered. 

Inputs and Key Oi^nizational I^isions 

Conceptual Issues 

The environment is the source of needed resources and infomiation. 
It is also the source of constraints to goal achievement. The myriad inputs 
coming into the system are sorted and prioritized for use by governance 
and management, resulting in critical revenue. peiT^onnel and access 
(kcisions that influciKre implementation. 

Rei^enue decisions govem the flow of money and other resources into 
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and through the system. In JTPA, federal alkxation formulas determine 
nature of monetary resources, but SDAs are permitted to supplement 
JTPA funds with state and local revenues. 

Personnel ckcisions govern the flow of specialized knowledge and 
technical expertise into the system, TTiese decisions affect the quality of 
personnel, which dirw:ily influences program success or failure (Fran- 
klin and Ripley 1984). In JTPA, for example, managerial, service 
delivery, contracting, information system, and monitoring capability are 
needed. 

Access decisions in JTPA govern the flow of program participants, 
employers, and educators into and out of the system. Given the limita- 
tions on funds, only certain individuals among those eligible for the 
program will enter the system and be exposed to its interventions. This 
affects the nature of implementation and influences a program's "dis- 
tributive'* outcomes — that is, the outcomes experienced by different 
groups of clients, such as women *'s. men, minorities vs. nonminorities. 
youths vs. adults. Contracting decisions infli'^ni^ which employers and 
educational providers will participate in the .system, be coordinated with 
it, and be affected by that relationship. These arc examples of the kinds 
of input and decisionmaking variables to be studied in examining 
implementation. 

Measurement and Methodological Issues 
Measurement Issues 

Those resources with the greatest potential influence on implementa- 
tion and outcomes need to be defined and indices for them developed. 
Variables involved in funding, such as monetary vs. in-kind contribu- 
tions, must be (kfined more specifically. The loan of staff or equipment 
from a slate agency may supplement federal funds in a significant way. 
Pinning down different kinds of nonmonetary resources is, therefore, 
important. Job qualifications for staff must be specified in terms of 
^ucation, experience, and compatibility with political and organiza- 
tional agendas. 

Ttie characteristics of the personnel actually hired and assigned to 
various responsibilities must be broken down into useful indices. Both 
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fonnal cl ..nt access policies, such as eligibility requirements and recruit- 
ment rules, and info.-maJ access pmctices. such as the way in which staff 
handles outreach and intake functions, must be defined. The division of 
labor within the system, the struciure of decisionmaking and the nature 
of revenue, personnel, and accc;,., decisions must also be defined opera- 
tionally and studied over the course of the program. In many instances, 
the evaluation research literature will not provide substantial help, and 
tte evaluator will need to take an innovative approach in defining these 
variables for study. 

The relationship between inputs, revenue, personi^l. and access 
decisions and other components of the organizational system must be 
defined. Since resources limit possibilities in tte ccMiversitm process and 
provide a context for maximizing the use of resources within this 
process, relationships between resource allocation/utilization and ele- 
ments of the conversion process are an essential focus in process 
evaluations. 

Methodohgii al Issues 

What happens to monetary resources is somewhat simpler to track 
quantitatively than what occurs with the acquisition and use of personnel 
or the recruitment and selection of clients. Focusing on relationships 
unong these system parts and other components requires a combination 
of exploratory and descriptive research designs and a variety of data 
collection and analysis methods. 

For example, management information systems can be useful sources 
of data on participant access. A random sample of clients can usually be 
drawn from a particular historical cohort to determine typical outreach, 
intake, appraisal, and service assignment patterns. However, few JTPA 
information systems record information at the beginning of tlw service 
delivery chronology— particularly at outreach, recmitmcni, intake, and 
eligibility determination points— where the initial and more subjective 
access decisions are maife. 

Some of the more significant characteristics of the attitudes and 
behavior of staff, other providers, and clients regarding access arc not 
retrievable from information systems, program documents, or client 
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files, and niusi be inferred from observaiions and discujusions. or tmtn 
interviews with these individuals. Hiere is an extensive liieraturc on 
personal, telephone, oxui mail survey rcsearch that can aid the evaiuator 
in this task (Dillman 1978; Fowler 1984; Yin 1984). 

Because different forms of data collection will yield a combination of 
qualitative ;md quantitative information, analysis methods will nev^ssar- 
ily involve both statistical and nonslalistical techniques. As in all 
qualitative analyses, the goal of inductive inquiry is to discern underlying 
patterns in the data {Patton 1987), These patterns and the results of 
quantiuitive analy.^s, together, fomi a basis for understanding what is 
happening in this component of the system, and for determining the fit 
between this and other parts of the system. The integration of this diverse 
information must be guided by the specific research questions directing 
a particular evaluation. 

The Conversilon Pmress: Mission 

Concej^ual Issues 

Goals 

The mission component describes the goals of the organizational 
system. In JTPA, Congress established thn?e common goals for all SDAs 
rcgafding "economic disadvantaged" youth and adults: to increase 
employment, to increase wages, and to reduce uclfare dependency. 
However, Congress also granted SDAs the di*«:retion to tailor these gcKils 
to local employment and training problems. Within the legislative 
mandate, each SDA develops its own set of goals in response to 
cnvirtmmental constraints, the agendas of PIC members, and other 
eonsicterations. Once these goals are formulated, the objectives that are 
subsumed under each goal are specified. These serve as 'indicaton^" that 
inform governance and management about the level of SDA goal 
achievement. 

Because SDAs have considerable autonomy in developing goals, the 
directives driving the program vary across SDAs. Studying implemen- 
tation within a panicular SDA organizational system, or across SDAs, 
tl^refore, requires careful identification and description of organiza- 
tional goals, or the mission of the SDA system. Without this important 
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definitional step it is virtually impossible to investigate the influence of 
a given SDA*s distinctive mission on other compt>nems — particulai'y 
the work compom^nt — on rther pans of the system, Oi on outcomes. 

A major task in studying goals is distinguishing between nuinifest rnd 
latent gpah. For analytical purposes we can think of goals formalized in 
legislation or in written policy statements as a system*s manifest goals— 
i.e., those goals presented to the system's environment as its primary 
guiding principles. These may indeed be the goals that have the greatest 
influence over the system's activities. Given SDA autonomy, however, 
other poweri'ul agendas, or latent goals, may actually lake precedence*. 
**hese less tangible goals may rcpa*sent the major intluence on action* 
Oi^ani/.ational development inevitably results in the emergence of goals 
idiosyiK-ratic to a particular system. 

A second task is establishing how priorities are set among these 
manifest and latent goals, and what these priorities are. This is essential 
in understanding decisionmaking prtKesses, other aspects of the conver- 
sion process, and the significance of a system*s outcomes. For example, 
in JTPA clients must ultimately be matched with employers in Ic^ral labor 
markets. The emphasis an SDA places on employers vs, clients often 
varies. This is sometimes apparent in missior statements. For example, 
the two goal statements below are taken from the files of two SDAs. 

CUent4)ri€nt€d Mission 

To provide comprehensive employment and trairiing services 
required to prepare and place eligible SDA residents into subsi- 
dized employment. Specific emphasis will be placed on selecting 
employment and training opportunities that will increase the 
earned income of clients ;md will result in secure, full-time 
unsuhsidized jobs. 

EmployfMJriented Mission 

To assist local businesses to si>ive employment-related busi- 
ness problems, allowing both business and individuals to in- 
crease productivity and profitability. To support local economic 
development efforts by the preparation of low-income, unem- 
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ployed area residents as a workforce for new or expanding 
business and induslry. 

These priorities move program implementation in quite different 
directions. 

A third task is to determine how internally ccmsistent or conflicting a 
system's set of goals may be. Siwnetimes a system's manifest and latent 
goals arc similar, or at least compatible. In other ca.scs they may be in 
continual competition or severe conflict. An SDA may have a manifest 
goal of providing comprehensive services, but a latent goal of providing 
lower-cost services that meet employer needs. Or SDA goals may be at 
odds with suboNitractor gi>a]$. 

SDAs t>T5ically contract with a variety of organizations — public 
schools, private industry, the employment services, private sector train- 
ing organizations, and community-based oiptnizations. An SDA sub- 
contractor may purport to share an SDA's goals, but because subcontrac- 
tor organizations frequently deliver JTPA services as only one arlivity 
among others, agendas independent of the SDA may be the primary 
influence, and these agendas may or may not be compatible with the 
SDA's. The less dependent on the SDA a subcontractor is for resources 
to survive, the more its goals may differ, and the further its implemen- 
tation of the program and performance may stray from what the SDA 
intended. Also, the actual goals of the PIC may not be compatible with 
tfwse of the SDA's administrative agency. 

A fourth task is to understand how the system's most important 
goals—those actually being pursued—are influencing the methods u.sed 
to achieve them, that is, how goals affect means, or the work component. 
Some process studies of client selection and service assignment practices 
in SDAs have suggested extensive "creaming" of job-ready clients, 
resulting in a neglect of the hard-to-serve (Walker 1984, 1985: Orfield 
and Slessarev 1986). This has been linked to tl» mandated use of 
performance standards, which some studies suggest force SDAs to 
choose between conflicting goals— serving the disadvantaged vs. meet- 
ing standards — with the result that SDA missions have sometimes turned 
toward the latter in order to secure incentive bonuses. 
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In summary, the relalicmship between fomially articulated g(^ls and 
those which must be inferred from orgmiizational activity, and the level 
of consistency between these *^ts of goals, have a significant effect on 
other components of the system. They condition judgment of the sys- 
tem's compliance with a program's intended implementation strat^y 
and the efTcciivcness of that strategy in producing desiird outcomes. 

Most organi/iitional systems engage in planning to develop their go^ls 
and define the means for achieving them. Hk: plan resulting from this 
pxxicess is based on three kinds of planning decisions: ( I ) who will be 
given the program's treatment, or service intervention; (2) what the 
nature of the treatment provided will be; ^d (3) how funds will be 
distributed in order to deliver the treatn^nt. 

In JTPA, decisions about who is to receive the program's service 
strategy determine the general target group of this strategy* and the 
specific subpopulalions of clients to be exposed to the treatment. Hie 
Congress establishes general eligibility and target group requirements, 
which act as screening and sorting mechanisms for assuring that services 
are provided lo those who need them most. In some SDAs, these 
mechanisms may fulfill this purpose (Walker et al. 1984, 1985). How- 
even client populations involve a range of **job readiness'' or "need,'' and 
as discussed earlien the pressures of pcrformaiKe stamlante sometimes 
QCt as powerful incentives to serve those whom staff view as most likely 
to ot^ain the higher paying jobs. This phenomenon is an important 
element to be studied in gaining knowledge of the actual practices of the 
organizational system. 

Treatment decisions direct which genera! set of services are to be 
receive by clients, and which mix and sequence of seivices is consid- 
ered more appropriate for one subjwpulalion than for another. Distribu- 
tive decisions regulate the flow of re«)urces to the service delivery 
system and to clients. Tliey determine what combination of n 
personneK equipnuiint, and support will go to whom. Ttese are p 
d^tsions that involve weighing competing claims against an agv 
resources by various interest groups, each seeking decisions in then 
favor (Franklin and Ripley 1984). 
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In JTPA these may be sulKoniractors or orgaiii/ations in the pro- 
gram's environment. Studies suggest that failing to i^lisfy such groups 
has often led SDAs to engage in defensive maneuvers rather than to seek 
an integration of ?^rvices related to local needs. In practice. SDA 
governance tries to strike a balance among competing demands — i.e., the 
service needs of clients and employers, jH^rformance objectives, costs, 
and politics. Decisions are also made about entering into coordination 
agreements with other organizations to increase the efficiency of service 
provision or expand the scope of available services. 

In studying planning, it is important to remember that plans may 
mrurately reflet intentions, but organizational systems characteristi- 
cally remold plans in the process of carrying them out, and sometimes 
plans are deliberately circumventetl. Both the attributes of formal plans 
and the characteristics of activities flowing from these plans are signifi- 
cant subjects of study. 

Measurement and Methadohgical Issues 
Measurement Issues 

Distinguishing l^iween miinifest and latent goals inevitably involves 
qualitative information. Tte evaluator can identify manifest goals from 
legislation and recorded policy pronouncements, but these are frequently 
abstract and require considerably more sprcification by the researcter. 
Answering questions such as the following may be of assistance in the 
measurement pnxress: 

1 . What is the source of a particular goal statement, and what priority 
in the hierarchy of goals does it enjoy? For example, it is important 
in judging the influeiKc of manifest gc^s to know whether the 
statement is mandated in tte program's legislation, is unique to the 
organizational system's top decisionmakers, or is an interpretation 
of the legislative mandate by administrators. 

2. What information can be inferred frcnn a particular gc^l statement? 
For example^ the evaluator may want to classify goal statements 
along the following din^nsions: the content of tl^ statement, who is 
to be most affected by the goaJ^ what outcomes are implied in the 
goal, and how the goal articulates with other goal statements. 
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Even mort! subjective, latent goals aa* exceedingly diOlcuh lo define 
and mcasurc, Infomialion must be obtained from the self-report of 
decbionmakers or infcircd from oi^anii:ational activities. Consider the 
goal of "increased wages" fw clients^ The evaluator can draw insights 
about the meaning of Ihis gixil by studying resource alkxration. If an SDA 
allocates resources so as to meet both federal performance standards 
regarding wages and organizational goals that are sensitive to the 
educational and training needs of certain clients, or if the SDA foi^iKs 
the possibility of meeting or exceeding standards in order to provide 
richer pre-employment services, this difference in allocation priorities 
will help define the mission component. This illustrates how interrelated 
the infomialion is that must be developed by studying different compcv 
nents of the system. In this case, the evaluator seeks clues about the latent 
mission from infonnation about the utilization of system inputs, the 
nature of governance and management, and the work Ciiniponent of the 
conversion process. 

Measuring goal consistency is essential but equally difficult. Consis- 
tency can be defined in terms of whether different goals imply different 
means, or dtlYerent courses of action. Attributes of goals, such as 
compatibility, competition, or conflict, must also be defined more 
precisely. TTk? evaluator must usually develop his or her own definition 
and indices, tailored to the program being evaluated. 

The planning process may also pose measurement problems. Some 
aspects of planning are relatively simple to measure, such as the compo- 
sition of planning b^-xlies, the chariH:terislics of planning policies and 
procedures., and the content of plans. Others are more dilTicult. For 
example, a chief concern of governance in most organizations is the 
difference between planned and actual performance. However, in some 
social programs govem^mce and staff develop a written plan as a 
prerequisite to otHaining funds, but once these funds are awarded* 
management may pay linle attention to the plan. In this case, the most 
difilcult measurement problem is to classify actual activities in such a 
way that they can be compared against the plan. Some of these activities 
are captured by quantitative data in information systems; others must be 
inferred from observation and interviews. 
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MaluHlologh al Issues 

Clearly an exploratory or descriptive design is most appropriate in 
learning about the mission component. An analysis of documents, 
observational techniques used at meetings of decisionmakers, open- 
ended interviews with selected policym^ers and administrators, and a 
study of organizational activities* such as the characteristics of the 
planning process, are examples of useful data collection techniques. 

An analysis format must be developed in advance of qualitative data 
collection, based on the research questions guiding the evaluation and the 
variables selected for study within tlK? missicni component. Once infor- 
mation is collected from different sources, tte challenge is to integrate 
these data to provide an accurate picture of the key goals of the system, 
which, in turn, affect the means selected for goal achievement and how 
well the program's implementation strategy is carri^ out in practice. 

The Conversion Process: Work 

Canceptml Issues 

5''Mdying the means a system uses to achieve its mission reveals the 
distinctive characteristics of the actual mode of program implementa- 
tion. As indicated earlier, "means*' refers to the activities a system 
engages in to achieve its goals and objectives. To illustrate some of ihe 
conceptual issues involved in studying the work component, we will 
focus on subcontracting and service delivery. 

Suht'ontracting 

An important aspect of identifying a system's means for goal achieve- 
ment is deciding what functions or services will be performed or 
provided by the SDA, PIC, jKiministralive agency, or oi^anizations 
undercontract to one or another of these entities. In JTPA, subcontractors 
arc an important element of the organizational system. All three of tte 
planning decisions discussed earlier are reflected in the nature of subcon- 
tractmg entities, the criteria guiding contracting arrangements, and 
subcontracting processes. Since subcontracting introduces another set of 
organizations into the system, it increases its complexity. The emphasis 
on performaj^-l^sed contracting in JTPA further complicates the work 
component. 
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Subccmtnicting, however^ may increase the efficiency of re?H)urce 
allocation by giving SDAs ite opportunity to choose among competitors. 
It may allow a PIC to reap Uk; benefits of succe^ful program perform- 
ance while spreading tl^ risks among a number of organizations. It may 
spur ths SDA to make specific decisic^ about which services are to be 
providol and to which target groups, and how service delivery is lo be 
coordinated. Or accountability may be the paramount coiK^m in award- 
ing contracts, that is, transferring partial accoimtability from a council or 
administrative agency to subcontractors (Walker ct al. 1985), The 
organizational politics of subcontracting is, therefore, an important area 
of study in process evaluations. 

There are usually two basic kinds of contracting for services in JTPA: 
(1) market subcontracting and (2) subccmUBCting by Junction. In the 
former, the subcontractor organization f<x:uses on a particular client 
group, or geogra{^icai region^ such as youth, women, minorities, or 
particular counties within an SDA. In the latter, funds are divided among 
subcontractors according to a particular service or occupational area, 
such as the provision of on-tl^-job training or training in the area of 
financial services. Both the choice of a subcontracting mode and the 
selection of subcontr^tors reveals important features of the system's 
goal achievement strategy. 

In a decentralized system, accountability for performance is trans- 
ferred downward through the vehicle of subcontracting. TTirough a 
request-for-proposaJ process, potential subcontractors are informed about 
SDA expectations: number of clients to enroll, services they should 
receive, and number of clients expected to achieve the desir^ outcomes. 
In subcontracting by function, tl^ P?" jr administrative agency is 
typically accountable for the access, treatment, and distributive (feci- 
sions incorporated in tte contract, while tl^ subcOTtractor is typically 
responsible for meeting the perfwrnance standards in the contract 
through its own similar kinds of decisions. 

In market subcontracting, organi^tions may be given greater fr^om 
to decicte how their training fumis are to be allocated among tl^ services 
authorize in the contract* A chief task of process evaluators is to suidy 
the planning decisions reflected in contracts, the extent to which they are 
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honored in terms of subconiraclor aclivities, and how these mlluenccs 
affect the overall implementation of the prt^gram and its outcomes. 

For example, the initial decisions sut^^uently iiKluded in a contmct 
may have been problematic or fXKjrly communicated. Or, subcontractors 
may have been only wciUcIy monitored for compliance with the contract, 
or may have simply ignored SDA expectations for peiformancc. Key 
issues if) contracting anc (1) the level of coordination and cooperation 
among subcontractors, and between the PIC or administi^ative agency 
and the subcomractoni; and (2) the extent to which SDA goals are 
successfully implemented or gradually displaced by the changing power 
relationships within a decentralized service delivery system. Competi- 
tion and dissension in this area have often subverted a program's 
intended implementation strategy (National Alliance of Business 1984; 
Walker ct al. 1985; Orfield and Slessarev 1986). 
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The Senice Deliven- System 

Within the iargcr organi?:ational system, the service delivery system 
implements the program's service strategy. TTiat is, the service delivery 
system is the part of the model that actually provides services to clients. 
The service strategy is a set of services, and sometimes subsidies. For 
example, the provision of on-the-job tmining to clients involves subsi- 
dies to employers. In both instances, the intervention is expected to 
change the behavior of the recipient; the client is to increase his or her 
v^ork experience and occujuiiional skills, while the employer is to give 
more attention to hiring the disadvantaged. However, implementation 
and service strategies rarely work precisely as intended. On-the-job 
training provided by some employers may be little more than unsuper- 
vised "make work.*' Employee may lay off otl^r, more highly skilled 
workers in oitier to acquire subsidized workers. The study of prc^ram 
implementation must capture these nuances. 

In virtually all swial programs, clients receive services in a standard- 
ized sequence of steps, frequently termed the client pathway (Patton 
1986), Each step, from entry to exit, is a necessary condition for the one 
that follows* Each involves certain underlying assumptions that link this 
chain of activities together. Figure 4.2 illustrates JTPA's client pathway. 
Table 4.2 suggests common assumptions. 

The way stations on the pathway outlined in figure 4.2 represent 
decision points in the movement of clients through a progression of 
activities, which ultimately expose them to the program's service inter- 
ventions and lead to the achi e vement of outcomes — outreach, recmit- 
mnni, intake, eligibility determination, assessment, plan development, 
service assignment, case management, referral, placement, and follow- 
up. Each step in the chronology needs to be studied and analyzed with 
respect to the following fetors: the nature of tl^ policies asKl manage- 
ment directives guiding what happens to clients; rewardt and sanctions 
for sta^ and clients in adhering to tf^se guidelines: staff and client 
attitudes arid behavior, the attitudes and behavior of other key individu- 
als interacting with clients within the pathway, such as employers and the 
personnel of other programs to which a client may be referred; and 
relationships between staff, clients, and others involved in the pathway 
with those outside tl^ organizational system. 
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For instance, studying recruitment may involve determining the effect 
of scarce resources on the use of outreach services, and ultimately on 
th(»se served by a program. Important to study is the nature of entry into 
ihc service delivery system, i.e.. whether a client can enter the system at 
a single point of contact in a single visit, or whether several appearances 
at numerous offices are required. 

Intake staff often act as system "gatekeepers." making discretionary 
decisions about access within organizational goals, planning decisions 
and clients' circumstances (Nagi 1974), An activity as apparently 
.straightforward as eligibility determination can be complicated by con- 
flicts in system goals or staff prejudices and management preferences, 
which influence the kind of clients who subsequently experience the 
interventions and 'dre expected to produce the outcomes desired 

Clieni appraisal and service plans hold another key to the way in which 
pmgram resources arc used to achieve organizational missions. These 
latter activities arc unusually vulnerable to the personal ideologies and 
agendas of staff, and the self-assessment and communication skills of 
clients. The actual assignment of clients to services may be conditioned 
by screening processes designed to determine client motivation, and on 
assessments linked to satisfying unmet performance standards. The 
actual provision of services may be fragmented across a number of 
subcontractors, with the possibility that clients may fail to experience a 
coordinated mix and sequence of intended services. Or, the assignment 
of clients may be affected by a "pigeonholing process." which classifies 
client's needs within a standardized set of categories (Mintzberg 1983a). 

This standardization of aspects of the pathw iv simplifies service 
delivery and conserves rcsotrccs, avoiding the customization of deci- 
sions at each step. However, predetermined diagnostic tools are not 
perfect and frequently lead to faulty judgments. The way in which 
a.ssessment and other semidi.scretionary decisions are controlled by 
prcdevelopcHi fonmats has an important effect on client selection, what 
services clients receive, how they are treated in the pathway, and their 
outcomes. 

It is also important to analyze changes in clients as they move through 
the pathway. Attention should be given to whether a service intervention 
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actually reduces or eliminates a client's needs, and how much service 
integrity is involved. »hai is, to what extent the client received the services 
formally prescribed in tl»c servio; plan, or rec^eivcd fewer or less 
intensive ones, or ones subst ilutcd by the subcontractor for those planned. 

In summary, these are only a few of the many factors that can be 
studied in focusing on the woii compotHjnt of the organi/aiional system. 
Again, Ihe emphasis in a given process evaluation will dejxrnd on the 
research questions that direct ihc research. 

Measurement and Metkadohgical Issues 

Measurement Issues 

Given Ihe bn^id range of variables involved in studying the work 
component and its relationship to other parts of the sy slcm, it is clear that 
some factors pose minimal measurement problems, some pose many. 
For example, developing categories lo measure the lypc(s) of subcon- 
tracts in an agency is often relatively simple. However, measuring the 
variation in monitoring proceduitjs. compliance protocols* and enforce- 
ment of penalties across ly|:H:s can he difficult and often can only be done 
qualitatively. 

Data collected for n:poriing pur|xises can assist the evaluator in 
measuring the characteristics of the service delivery system. Manage- 
ment information systems selectively describe the client pathway, which 
permits descriptive analyses of client flows through the pathway. Unfor- 
tunately, few MlSs contain information on program applicants or precise 
descriptions of the particular services and subsidies provided to individ- 
ual participants or to different tai^et groups. These systems rarely 
contain information < m others affected by a prognun* such as employers. 
Also, some MISs do not permit easy access to a client's entire program 
history, focusing instead on the production of aggregated data. 

Therefore, although quantitative measures for some variables are 
readily available in MISs, a combinaticm of indices routinely used in 
information systems and data from other sources—both quantitative and 
qualitative — will be needed in studying the service ctelivery system imd 
its relationship to other parts of the conversion process. Measures of the 
service deli'/ory system suggested in the survey reseaa-h literature can be 
useful in supplementing data in information systems. 
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Mel/Kklohj^k al Jsxues 

Exploratory and relatively unstructured descriptive designs are the 
choices for 5itudying subcontracting. A review of documents supple- 
mented by interviews with key actors may be the most useful data- 
gathering techniques (Burslein et al. 1985). A formal for analyzing this 
qualitative information, consistent with the research questions, must be 
developed by the evaluator. 

More rigorous descriptive designs can be used in studying the service 
delivery system. Ekxrumenting what occurs at each major step in tte 
client pathway, and what validity assumptions are involved, is a first step 
in studying the service delivery system. Determining the nature of 
attritic^i as a cohort of clients traverses Uiis {^way, l^sed on figure 4.2, 
is important, sina; this identifies the points at which particular kinds of 
service delivery problems may be occurring. For example, in tracing the 
origins of a wagedifferential between male and female participanLs. the 
attrition patterns for the two sexes can provide significant clues about the 
sources of wage differences. 

Much of what is happening along the pathway is available for analysis 
in an MIS or case files. For greater information (ktail, interview or 
questionnaire techniques can be used with staff, clients, arai key others.* 
If response categories arc scaled, stati.stical analyses of this information 
can be performed. However, qualitative data from in-dcpih interviews 
can often yield more useful insights. 

The Conversion Process: Cooitlination 

Concej^ual Issues 

ImpLmentation studies can identify coordination mechanisms and 
as^s their effects on (Mher parts of the organizational system. There are 
numerous ways to cfxndinate activities and otter phen<»nena in organi- 
zations. One strategy is standardization, which often takes the following 
forms (Mintzberg 1979;: 

1. Standardization of outputs through the use of performance stan- 
dards. Although subcontractors may emj^msize different methods 
for producing outputs, performance staridards set jmrameiers that 
coordinate ^ivities. 
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2. StanJurdizatkm oj staff skills. Because professional skills arc needed 
in program organizations, staff members have considerable aul >n- 
omy and discretion in applying their skills* Standardizing these 
professional skills lets each staff member know what to expect from 
others^ which supports cooniination. 

3. SandardizatUm of tasks to be performed. For example, subcontrac- 
tors responsible for youth services may make the assumption that all 
youth have similar needs and, therefore, require the same services. 
While this assumption may not be valid, siandaniizing the service 
coordinates activities for youth. 

4. Infomial vimmmmvatum among staff performing a particular set of 
tasks. Sometimes ite most important coordination activities are the 
product of informal communication networks. 

5. Direct supenision and monitoriftg of staff. Coordination can also be 
achieved through management of staff. 

Tlirough strategies such as standardization, the efficiency of the 
organizational system is increased in terms of workflow and productiv- 
ity. Some issues that process evaluation can usefully address in this area 
are the following: 

1 . Matching of clients with employers, or the lahi^r exchange function. 
The key issue of interest is control over the matching process. If tl^ 
majority of clients find jobs on their own rather than through the 
placement efforts of staff, clients are in ccmtrol of employment out- 
comes. If most clients arc pl^ed through staff job-develc^ment 
efforts, staff have greater control over the matching process. Identi- 
fying these differences can alert the evaluator to problems in coor- 
dinating the l^>or exchange function. 

2. Relationships with subcontractors, and organizations in the envi- 
ronment to which clients tnay he referred for ser\*:ce While service 
delivery may be more effective when responsibilities arc distributed 
among a number of organizations^ ctecentralizatic^ requires in- 
cre^ed coordination efforts. Problems may occur when it is unclear 
who is accinintable fcH- what responsibilities, which occurs more 
often wl^n sulKontracting is carried out by function (e.g.^ when 
accountability for meeting a performance stmlard is distributed 
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acniHs many sulxronlraclors). Accountability, which involves effec- 
tive coordination, is stronger under market subcontracting. If only 
one organization is providing servitxs to the hard-to-placc, for 
example, it is clear who is accountable for the performance of 
pix^grams oriented to that group. 

3. Provision of multiple senives deiivered by different suheontravtors 
that require vtHndination over time. In this com?, clients must be ap- 
propiialely linked to a series of services in the correct sequence. Case 
management systems can increase the ctK>rdinat]on of a cliem*s se- 
quenc^c of service treatments* If the MIS stores the client's service 
plan, the MIS can be used to alert staff about changes in .service or 
service provider before their s^'hedulcd occunuicc. 

4. Coordination of intake \ ith other steps in service delivery. If intake 
is the responsibility of the administrative agency, and other organi- 
zations pmvide the servic*cs, it is important to know how smix>th and 
timely the flow of clients is from agency to contractors, and whether 
these organizations share validity assumptioas. IX> they defim^ 
clients' needs and tMirricrs similarly and agree on the kinds of service 
to be provided to a given individual? The evaluator needs to 
determine how problematic the relationship is between the two 
organizations and how this situation affects the referral network for 
clients. 

5. Displacement ofSDA f^mls. Subcontracting can distract an admin- 
istrative agency from its goals ny shifting attention to contracting 
ojkJ monitoring (Mim/berg 1979, 1983). Process studies can deter- 
mine to what extent goal displacement is occurring. 

All organizational systems must cootdtnate their activities to survive. 
Governance and management set the coordination agenda in the process 
of defining the organizational mission and decentralizing the service 
delivery system. The cvaluator\ task is threefold: to identify those areas 
of the system that are likely to nrquia* tte greatest coordination, to 
describe coordination policies and practices, and to examit^e the eiTec- 
liveness of coordination irtrategies in the ccMitext of other elem:jnts of the 
conversion process (particularly service delivery) and otl^r components 
of the system. Coordination issues clearly cut across all components of 
tl^ model. 
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Memunwwnt ami M^haial^cai Issues 
M^surement Issues 

Defining and measuring tte standardizattcm of client cmtcom^s, staff 
skills, and oigrjiizational tasks involve both qualitative and quantitative 
imiices. Tasks and sJdils r^uire the development of profiles that involve 
both kinds of data. Studying work flim\ volume, and productivity 
inevitably requires tl^ same cimbination of data. Som^ client outomies, 
on the oiher hand, are measured quantitatively in an MIS. 

Work flow can be measured in terms of client time, such as the average 
tin^ required for a client to move from oart point to am)thcr in the client 
pathway, and/or woilcer time, such as tl^ average time require of staff 
to complete paperwc^ or procedure associated with a given step in the 
pathway. Volume can be measured in terms of performance ratic^ related 
to costs, staff, and time. Costs can be measured by units of service m 
activities, siK:h as cost per counseling session. Staff pnxluctivity ratios 
can be related to workload standards, siKrh as number of clients per 
case manager. Time ratios can be based on the frequency of a program 
activity in a given lime frame, such as the number of clients screened for 
eligibility per month. 

Many of the variables and relaticMiships ctescribing different aspects of 
coordination can only be n^asured qualitatively, such as staff commu- 
nication patterns and the char^eristics of tte referral c^twork. TY^ 
evaluator can rely on the organizational literature in identifying key 
variables, but will frequently ne^ to develop his or her own defmitions 
and indices. 

Methodoiogical Issues 

kinds of research ctesigns ajKl the range of data collection and 
analysis methods discussed umter other parts of tt^ conversion process 
apply to tl^ study of coordination. An exploratory design may be most 
apjnt^riate wi^n ti^ SDA oiganizaticmal system is ccmiplex, with 
extensive decentrdization of responsibilities to subccmtractors, and 
wl^ little is known about the level of coordination. In simpler systems, 
or where studies of coordination have b^ conduct^ previously, 
evaiuaticms can use fairly sc^histicated de^ptive designs* Methods 
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will depend lai^ely on the ability to obtain and analyze reliable, &ppro- 
priate quantitative data. For example, in .studying coordinatiim between 
oiiganizalions within the system, and between the system and organiza- 
tions in the environment, data on the number and kinds of clients leferred 
tc^nmi these oiiganizBticms, and the distribution of services within that 
netwoilc can be «x;essed from an MIS. 

The Conversion Process: Sodid Climate 

Ctmceptual Issues 

The level of satisfavtion felt by aaff, clients, and others is important 
to the achieven^t of oi^anizational g(^s and is as indicator of the 
efTiciency of program implementation. Retaining high-quality, moti- 
vated staff is critical to oi;ganizati(»^ fura^tioning; therefore, i<temifying 
tite rea.sons fcM* dij^stisfactitm and turnover is a significant tai^ for 
process evaluators. The rcastms for staying or leaving may vary by job 
qualifications, position, amount of ttaining, and other factors. Factors to 
consider include tte degree of job satisfaction and compatibility with 
I»x>fessi{mal colleagues (and with the system) regarding values aiKi 
goals. Sources of stress that may encourage turnover inclwte the use and 
enforcement of perframance standards, perscm^ ci»iflicts over work 
decisions, and dis]^ties over what is "best" for clients. An employee's 
percepticm of tluj availability of better job o^^portunities, ix ixmwork 
fetors such as family ties, friendships, community relation^ps, ami 
finances may influence retention and turnover (Flowers and Hu^s 
1973). In performing this review, tte evaluator should also itetermine 
wheflicr mechanisms, such as an "open-door" style of management, exist 
for solving staff grievam%s as tl^y arise. 

In JTPA, employer satisfaction is essential to encourage, since the 
supply of training 'pb& is (kpemknt on good employer relations. In 
chapter 3. issues related to emplo>^r satisfaction are suggest^ in the 
discu^ion of onployer surveys. Examples are employers* overall satis- 
is^^m with tte program, with propam staff woriiing with particular 
clients, ami with the clients they hire or train. 

Client satisfaction is frequently studied in terms of the client's percep- 
tion <rf tte quality of tlw prc^ram's services, the manner in which tlwse 
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^rvices are provkted, ami the nstfure of smfT-client relatimdiips* A study 
of CETA classnM>m training pro^:ams illustrates the impcmaiKe of 
client attitudes for organizational goal ^hievenmit (Simf^on 1984a«b). 
Alttou^ clients gemrally nepcmed high satisfacticm with the prc^ram, 
nKHe detailed survey qu^tions revealed that the quality of training 
K«vi ved was ib« key reason for their satisfaction. Tte following training 
chanK^teristics nad an impmn: training that was l^^eu m a clearnrut plan; 
traming that felt 'ike employment; instructors who encouraged inde- 
pencteiK^ ami gave positive feedt^k about performance; the ability to 
communicate ami negotiate problems with the instructor fr^i^tly. The 
role played by tl^ instnK:tor also was a key factor. 

This study also suggest^^ Iwwever, that low client satisfactic»i was 
not positively associated with ncmcompletion of a program* Financial 
problems^ job c^>pommities« and a change in vocational goals were major 
reasons for dropping out. In f^t, a sut^tantial number of dropouts were 
quite satisfi^ with the program. In this sense, stud> ng the reasons for 
drqyping out has implications fcH* tbs snting and screening process at the 
beginning of the client }^way. Ttese findings also have implications 
for service design and implementation^ by either the administrative 
ageiKry or tl^ subcontractor. 

In evaluating satisfaction levels, factors within and outside tte wgan- 
i^aticmai system* siK:h as the actequacy of lesoinces to serve all eligiMe 
clients* must be stiKlied. Client i^ways may create dissatisfa^icm 
through waiting lists at intake, attendamre requirements in training, and 
other sorting n^hanisms that tend to ^{mate clients on the l^is of 
motivation and satisfaction. The actequ^y of job opportunities at t^e emi 
of program partictj^tion can also condition client satisfaction by feeing 
an organizaticmal system to focus on clients* shortcomings and their 
•^rej^ckaging*" to better compete for jobs. This '*refOTm** may produce 
high employer satisfaction but low client morale* 

Measurement and Methodahgical Issues 

Measurement Issues 

TTie definiti(m ami measurement of variables such as satisfaction, 
motivation, and nK)rale involve attitude measurement* a major area of 
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concentration in social psychology. Although thinks of attitudes as 
qualitative variables, the field of attituck scaling involves quanttfication 
of respoase categories in highly structured survey instruments, such as 
qt^tionnaires and interviews, which are cksigned spcctflcally to obtain 
information on the values, beliefs, and emotional states of lesponctents. 

Attitude scaling has often been confiiwi to outcome studies, partly 
because of their more rigorous nature and longer research history. 
Process studies have tended to rely on qualitative infom;ition obtained 
by more open-eiuted sinveys. The substantial liteniture m aititucte 
nu^urement, however, offers the process evaluator an opportunity to 
use a combinati(»i of indices, which can expand the knowl^ge Iklsc on 
program implementation (Edwards 1957; Upshaw 1968). 

MethfHiologu al Issues 

The collection and analysis of information cm attitudes. reg^Iess of 
the level of rigor, are complicated by the ii^vitable measurement errors 
involved in self-reporting. All surveys (tepend on respondent horesty, 
openness, ami insightfulness. The more sophjsticai«I data collection and 
analysis techniques us^ in studying attitudes tend to fon^ choices, and 
may not permit the respondent to fully express the trv« range or intensity 
of feelings, to explain the context in which they occur, or to provide the 
reascms for feeling them, 

Open-encted survey questions afford more opportunities to probe for 
this kind of detail. Statistical analysis of scaled data and content analysis 
of more informally collected data have different tradeoffs. Again, the 
status of previous knowledge about tte research questions of interest, 
and tl^ willingness of the evaluator to accept different degiws and kinds 
of bias, will determine the methodology used in studying this aspect of 
the conversitwi process."' 

Outcomes and Imp^t 

Conceptual Issues 

Several kmds of outcomes can be produced in il^ JTPA conversion 
process: (1) mandated outcon^s for clients, in i^icular, increase 
employment and earnings and rediK:cd economic depetulency; (2) poten- 
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tial outcomes for employers, such as lower employee recruitment and 
tnuning costs; (3) required outcon^ fcnrthe or^ganizationai system itself, 
in teams of meeting performance standards, achieving local gc»ls, and 
iiKrreasing the efficieiKy of prc^ram opmtitm; and (4) possible out- 
ccmies for other systems. siK;h as reduced carloads and costs in the 
welfiare and/or unemployment imurance systems. 

As iiuiicated in table 4.2. achieving Icmg-run im|^s that go beyfMid 
inmiediate pn^iun outcomes is the ultimate aim of JTPA. Outccnnes for 
clients measuied at follow-up points beyond program exposure may not 
be as positive as outa»i^ at program c(»npleti(ni. The effect of the 
program on its environment may be minimal. For instance, the program 
may nm have reduced unemployn^t in ti» commtmity . Hie number of 
ecfflUNiiically (tependent clients making (femands on <Hli«r programs 
may not have been decreased. Referral n^tw^ks among service provid- 
ers may not have experiemred better coordination. ImplemoitaticMi may 
not be the source of problems, but the process evalt^or will want to look 
for pcM»»ible relationships between compcments of the organizational 
system and long-term impacts. 

Memwemnt end MetMt^iea! issues 

Outcome measurement has b^ covered m chapters 2 and 3, in terms 
of (Hilcomes at program exit and outcomes at follow-up points. The 
impact of the environn^nt on tte |»t>gram has been recxfgflixcd in 
outcome studies, but (kflning variables and develc^ing mea^res 6c- 
scriNng the imp^t of tl^ progr^ on its environment have tiisditicmally 
been r^glect^. largely b^use of data accessibility and the difficulty in 
sorting out the separate effects of implementation ami service strategies 
from f(Mices in the envimin^nt. In most cases, estimating a propam's 
effects on the enviroranent is beyond tte scope of process evaluations. 

Feedbrck: The Perforamnce Contnri Syirtem 

ConeefimA Issues 

Some kiml of mechanism for keeping governance and management 
continually informed about progr^ outcomes and ti» broaifer imp^ts 
of a program is necessary if the organizational system is to be fu%-tuned 
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or modified to increase efficiency and effectiveness over time. The 
performance amtrolsysfem is designed to a:complish this task (Minlzbcrg 
1 979). It consists of two parts: ( I ) a plan describing desired outputs and 
impacts within mandated performance standards ami local goals, and (2) 
man^ement cfmtrols to determii^ whether tiK plan has b^n achieved. 
TTk performaiK^ (^ntrol system is, therefore, a monitoring ma:hanism 
Ms&A by governance to manage program and oi^izational performamx. 

In a highly decentralized system involving multiple subcontractors 
with ccmsickrable autonomy, acfaninistratms usually lack the authority, 
and sometimes the rescnirces, to control subcontractors* me^ for pn>< 
ducing outputs. Tlwreforc, mtniitoring str^egies are limitnl to diking 
m a narrow range of outfnits that can be quantified in an MIS, or 
relatively easily coded and analyzed from program forms or 
qiKstionnaires. 

To be effective, the monitoring fuiKtion has to involve rewards for 
compiiai^ with the plan and sanctions for nc»Konformance. In terms of 
state-level performance control systems, the reward is increased funding 
to the SDA, which can be us^ for its ovm purposes within state 
guidelines. The sanction is compulsory technical assistance provide by 
the state to the SDA in order to identify and correct performance 
IMoblems, In the case of the SDA monitoring system, the reward can be 
reiKwal of a subc(mtnK:tor*s contract or provision of a bcmus ot increased 
aut(momy, and tl% sanction can be technical assistance provided by tte 
PIC or Miministraiive agency to ihe subcomr^tor, or cancellation of a 
contract. 

Monitoring systems represent a way to reinforce an oi^ganizationai 
system's a)mpliance with goals and a means for judging tl^ system's 
success in a«:ompli^ing its mission. However, monitoring can also be 
intrusive ai^ diiiruptive, creating tension between the monitor and ilw 
monitored (Minuberg 1979, 1983). Perfomiance control systems that 
extend beyond basic oversight — checking on the meeting or exceeding 
of performance standards, verifying expeiKliturcs and confonnance with 
Iwsic rules and regulalicHis — can also consume excessive administrative 
resources and redwue staff morale. 

For example, technical assistance in JTPA sometimes involves a 
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managerial tc^rhnique, svdi m hnpc^ing mem goals or iiKreasmg t)^ 
intaiBJty ami frequoKry of irKmitcHing. At tte SDA level, this inoeased 
adminis^raUve pressuie frequently engen^rs lesistamx and can redirect 
subcontracmrs towanl cDmpliance-<memed ^tivities, which may fur- 
Iter interfere with successful overall performance. SDA staff may be 
driven to ignore the social consequences of iheir decisions in onier to 
meet performance niandards, which can result in a neglect of tho% most 
in iKwi of service. If tl^ competence of subc^nitractor steiff is the 
problem, tougter mmitoring techniqi^s c^ be a u^te of tl^ SDA 's 
ministrative energy* 

Mcmitoring is also ineffective if tlK^ perfonnance ccmtrol system does 
not include a committnent to reporting^ or providing feedl^k to govern- 
ance and management. The quality of monitoring and reporting affi^ts 
decisions about modifying the system* Faulty fe^back can lead to 
flawed changes. In geroral, although performanre control systems are 
e^^ential, overdeveloped systems can stifle innovaticm, encourage con- 
servative progrmi management, am! overemphasize easily quantifiable 
ecoiK>mic outputs, such as earnings, while downplaying less easily 
measured social outcomes, such as increased employab^Hty* 

The use of an MIS may contntnite to ttese potential proble.ns. MISs 
appear to be tte most efficient way to manage such information, but, in 
fact, reliance on them restricts the content of monitoring plans and limits 
monitoring strategies. As a result, govemarKre and mana^ment may 
misjudge the underlying causes of performaxKre problems. Hierefore, 
actions that the system's leklen^p takes to ameliorate these problems 
may be t»$ed on mislei^lingly sparse or unrealistic information. 

Measurement and Methadohgical Issues 

As menticHwi eaHier, a key task is classifying tl^ content of written 
nK>nitoring plans ami i^iocedutes, which includes the following dimen- 
sions: (1) goals and rationales, (2) alternative courses of acticm to be 
taken regarding compliance, (3) rewards ami sanctions, am! (4) informa- 
ticwi to be reported to governance and managen^t The san^ process can 
be appli^ to the monitoring strategy pr^ticed by the £^ency, which 
irxrludes (1) activities used to itkntsfy various forms of noncompliance, 
(2) aciicms taken to correct compHaiKre problems, (3) activities involve 
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in rewards aiKi sancticms. and (4) infonnaiicNi mually report^ to 
governance and management about thejse activities. Discrepancies be- 
tween the written plan and actual practice may point to potential 
l»eakdowns in tlw operaticm of tte feedback »vstem. 

Monitoring ami reporting involve the judgment of information about 
im^ram mitcomes and the impact of tlwsc outcomes on a pit^ram's 
environment, and the communication of this information to those re- 
sponsible for adjusting program impIen^ntaticHi and performance. A 
basic question is whether administrators are ^mially receiving and using 
inftmnalion from the performamx control system in (tecisionmaking. 
Descriptive <tes;rjns and qualitative methods are most usefiil in answer- 
ing this question. In-depth interviews of an SDA's monitoring staff and 
i^isiomnakers in the PIC, administrative, or contract agencies can 
supplenK^nt content analysis of monitoring a«J reporting documents. 

Studying the **Fit*' Among Parts 

A» the beginning of this chapter, a model of an oi^ganizational system 
was presented to be used in evaluating program implementation. An 
underlymg assumptitm of the model is that organizational effectiveness 
and efficiency depend on the degree of integration, or consistency, 
among the system's parts (Lydcn 1975; Harrison 1987). Thus, after 
collecting information about program implementation, a chief task is to 
determine the degree of consistency among the following elements in the 
model's era version process: 

1 . Between tte missicm compownt and the environment. 

2. Between tl« mission am! work ccmiponents. 

3. Between the mission and coordination components. 

4. Between the mission component and the feedback system. 

Two other factors must also be asf ^ssed. First, organizational systems 
cannot perform well il ite mission, work, or coordination components 
are "missing" from tte conversion nrocess. For example, organizations 
without goals (a mission component) or a well-cfcrined client pathway 
(work compoivnt) often have poor perfomiam^ records. Infoniiation 
collected during the course of the process evaluation is used to make this 
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i^ises^nent.^' Secoml« the fiocml climate of an organizational system h 
an indicator of how well tte system is q>eratiiig. Thus, the (tegrce of 
satiHfmrticm ami tmstc^ in the f^ys^cm mast al%o be assessed, as discussed 
earlirr. 

TlKse are tte major, initial points of iiK|uiry« If the evi<tence suggests 
that inccmsistcncies exists or ttmt compoiHrnts from the ccmversicm 
{^ocess, or that staff morale is low, furtter data collection analysts is 
oft ?n necessary to discover tite underlying leascms for these (Hoblems. 
Process evaluations have distinct cycles: a macro cycle to assess the 
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overall operation of the oi^ganizatitmai system, and one or more micm 
cycles targeted at specific issues discovered at the macro level. 

Are the Parts Consfeitent?: An Example 

An example may clarify these points of inquiry. An evaluaior has just 
completed a process cvaluaticm to determine why female clients have 
lower placement rates and wages than male clients. The Txtsx task is to 
compare the infcwmation collected for the environment and each compo- 
ncm of the conversion process. TTki goal of this inductive analysis is to 
disco%t;r commtm i^ems by performing tte ccmsistency checks de- 
scribed above.'- The major results of this effort are organized to help 
compare findings across components of the model, as shown in table 4.3. 

In our simplified example, a of fit appears to exist between the 
missicm compwicnt and the work component. Given this source of the 
performarK^e problem, a second round of micro-level analyses could be 
targeted at discovering its likely cause(s). such as poca-ly designed 
screening procedures, inadequately trained intake staff, or simply that 
few females with work experience apply to the program. 

In summary, in the course of this investigation tte evaluaior has taken 
the following steps: 

1 . Reviewed the research literature to learn more about issues the pro- 
gram is meant to ^ress. 

2. Become thoroughly familiar with the program being evaluated. 

3. Developed a clear understanding of the r^arch questitms of major 
interest ami the use to be made of the answers. 

4. Collected inistworthv information on the separate influences and 
components of the system. 

5. Utilized applied research skills innovativcly in integrating this 
divene range of informmifMi. 

6. Relied on inductive judgments about "fit," qualifying the assump- 
tions underlying these judgments honestly and fully. 

In documenting this process in the final report, any alternative (but less 
convincing) intenMetations of the dat.. should be presented to dem«)- 
strate that a thorough analysis was conducted. RccommcndaUcHis for 
changing the parts of the system to achieve a proper fit among its parts 



Hvaluuting Pn^ram ImplenKiitation 291 

fAvmki also be included. In ourexample. pcrformamrc stamlards might be 
lowered to incieaiie tS^ c(HiHistcra;y of the mission and work omipmtents. 
In a^ition. new intake procedun^ might be installed along with better 
management cfmtrols to assure proper sei^^ion of clients. 

Did the ImplemenUitioi Strategiy Work? 

Osux the at»ve analy Us is ctmipleteU. the cvaluatoi- can examine the 
merits of the program's untterlying implementation strategy. As men- 
timted at the Siginnin^ .'>f the chapter, each social pn^m involves a 
theory regarJing progi^m implemcntaticm. TV actual implementation 
of the program may differ from the tteory. For example. tl*c public- 
private (wtnership in JTPA, which increased t\K role of business in tfie 
governance of employment and training programs, was intended to 
increase riient placements, but this might not happen in practice. 

In interfHieting the Hndings of process evaluati{»is. two overriding 
potential questions are. ttercforc, "Is the lheor>' valid, and docs the 
implementation strategy woiit?" (Scheicr 1981). In practice, pnx:ess 
evaluaitons tend to igiM>re ih: first questiwi and ccHKentrate on the 
second. If possible, process evaluaiicHis shcmld seek clues about both. If 
informatimi from an evaluation suggests that the theory itself is {NDblem- 
at'c, then improvements m: indicated in the social i<koIogy or policy on 
which the theory 's ba.sed. If the tl«ory appears sound Init the imptem- 
entation strategy does not seem to be woHcing as intended, then improve- 
ments in the way the progiam is organized and o}Krated arc indicated. 
Sorting out the information from process studies along these lines 
provides important ^luidance in planning change. 

ComrluHons 

A complementary relationship exists between process and im{^ 
evaluation, impact evaluation, which is pmrt of an organization's fi^- 
t»ck system, iktermines tlv results, or outcomes, product by tl^ 
pn^ram. However, by tlwir methods, impact evaluaticms typically treat 
prugrams as "black boxes"; what goes on inside the program (or box) to 
produce the results is rarely assessed. Process evaluations fill this gap by 
analyzing the processesi that produce program results. 



2sa 




^2 Evaliuuing Social PrdMenu 

Findings fipMn a process evaluation often provitte direction for ccm- 
ductiiig impact stmlies (P&tion 1987). A process evaIua:ion might be 
conducted to discover impoftant issues and develop hypotheses, which 
ane lest^ using tli^ grcMis impact modei in the previous chapter. For 
example, qualitative results may suggest that participants receiving a 
thcnough iweds assessment have better outcomes than ihoje who are mn 
assessed. This hypothesis could be tested in a gims impact model by 
including a dummy variable indicating whether the particijKmt received 
a needs assessmem. Shapiro ( 1973) provides an example of how differ- 
en^ betw^n quantitative md qualitative results were re!U)lved in a 
study of elen^tary education. 

In summary, thcnc are several characteristics of process evaluations 
that should be remembered. Compared to outcome studies, process 
evaluaticNis require that much more consiikration be given to tl« 
development of research questions and a study design capable of answer- 
ing U%m. Thh requires a much more deliberate election process for 
focusing evaluation activities on those influences and relationships of 
greatest interest ^nd relevance fcM- information users, within the con- 
straints imposed by a sparse research liieraiurc, the difficulty in defining 
and measuring some of tlw key variables, and the necessity of collecting 
qualitative data that are not designed for statistical iuialysis. 

Secfmd, in designing a process evaluaticm, a unique mix of data 
collection and analysis methods must be carefully selected to fit the 
research questions to be answered. 

Fmally. becau.sc of greater difficulty in measuring implementation 
variables and l^avier dependence on qualitative data, the process evalu- 
alor usually has more discreticm than the outcome evaluator. TTiis 
nKthodological fre^(»n has a cost, lK>wever, for it makes the task of 
integrating and inicrprer ;ng the range of information collected during the 
cour% of the study much more complex. 
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oigftimidk>ita} perfpiinaiK^ 

Z PjmMi f 19H7) pk^esemK mttbods fm-a&bcsjibig these rwics in evalu^mg pm^nsn smpkron- 
mkm^ which may be applied to the analyst of mt iyrgaxm^kma! system. 

3. DifTcmKes betweeii c^iamiti^ve «id 

in Item < I^K Scb«^am and Jatx4)s < 1979). ami Mintrbef^ < 1983a)« amm^ iH}K;fi>. 

4. If ttm qiiamiiaiiv^ appmdm m subject to mef^mremenl erm tami must irf t)^ art), the 
tnatei^ may be greater acmacy. In this cs^, qualitative data be tlie itmc Kcmu^ schocc of 
mfonnMioii. 

5. P^mofi ( 1W) describes a vjuriety <rf purposeful sampling strategies that may be on^oyed in 
evaltuaiom of pmgrDn tntfriemenimton. 

6. The posuhSe bias s^socia^ with ntNiprob^Mltty sampling; may be r^lumi by cnltertmg 
infonmtion fnm muhipte sair^i^ If ihs inforrmaion is cmsiskem acm&iii simipSes, a degree of 
ccnveri^m valklity n^y be obtained. 

7. S(Mne}m^gnunsarrtnq>fememeddbect}yasthe«siatec^ 

imder JTPA for dislocated workmopet^ed by states. wid|m^:fams » or Nmive An^ncwc^^eiated 
by the federal govemnKiu. &udying fvi^grsin tmpfeimtttatkm in t}»se instances invoK es the lanw 
^nmJ \m and relatton^ins but prinwy imemion is given to the tevci at wh^h the most 
hf^imin^ asyvcts of in^rtemcnMik^ ocnu-* tneattt^ the ollwr levels is *nhe cnvi imwrn." 

ft The llteranifr on studying fmgrani envinmments is span*. However, a usefui re» jrcr is a 
necem specie! issue of one of the Amencmi Evaluation Assodatio9i*s jounuik %vhkh is li^ in this 
chi|tter's refeirtKes. (See) C^mmi and Rf^»ate-Ck^ ( 1^: 40)). 

9. Simpson ( 1 9H6) piovkks (}oestk»maiies fw clients and employers in the JTf 'A 

la Simpssm { 1 986) and Gtiembowski < I9S6> piesent questioim for measuring client, employer, 
and staff saiisftnicm in en^k^rmeni and ttBming inogmms. 

n. Sometiniesorgsiiizatkmimtpl^zeonecoiiqtcmem 
velopment (Lyden 1975). For example, at start-up an oiganii»don*s chief concern might be to 
{mduce tangible n^tts. It. duts, puts emjrf^is on die ymk compcr^t, whkrh toer shifb to the 
coofdinaiicm component ffiicoid^ttsk»iKm»ims. Utter, attemicm n^ght shift totbe msssimicimiponeitt 
as t^iesikms aie raised abmit whether the organisation is wsking a difTeren^ in the txitskie world. 

12, A key concern in this and all analyses h the validity, or accuracy , of the findings. Pmtwi ( 1^7 ) 
explains how "Iriiingulation" tan be u?>cd to verify ihe mitial resuiu of thi* csca-inc. 
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Part III 



Organizational 

and 
Political Issues 
in 

Evaluating Programs 



State and local progmm evaluations take place in an envi- 
ronment vuli^rable to numerous political sav^ organizational 
pressures. TTw most bias-free and comprehensive evaluation 
plan may fail to yield information that is useful to and usable 
by (tecisionmakers in improving programs, if ihe implemen- 
tation of the plan is flawed organizationally. 

In this STOti(M), the authors examine an integral part of the 
state and local evaluation process, one frequently ignored or 
given CHily minimal attenti(Hi; that is, ihe development of a 
sufUcient oi^anizationaJ commitment and professional capa- 
bility to evaluate programs that support vigorous and useful 
program evaluations. 
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Usei^Centered 
Evaluation Planning 

Deborah Feldman 
Wiishingtan Employment Semrity DepartmcfJt 

Aft evaluation can no longer he seen merely as a creamre if the 
evaluator^s (m*n vhoosinff. Insofar as wr are dealing wiih the intent 
of being used in a ffoliiical environment, the t^hoive of the program 
to evaltmte and the types of piMi y t/uestions to he asked mast reside 
with the dccisi(mmakers. 

Ekanor Chclimsky 
Esvlaation Fimm 



Bas^ Concepts 

Thrce major complementary appmachcK to evaluating social pro- 
grams have been outlined in tte previous chapters* Taken as a wholc« 
tlKse ai^Daches are n^ant to provide a comprchemive view of evalu- 
atim pi^sibiii ties at tiKr state and local level . In order to supplement ihe% 
nK>re technical descriptions of evaluaticm iiK>dels* this chapter examines 
the practical evaluation planning issi^s tfiat crossHJut all three ap- 
pro«:t^: \mi to eH^ortively initime, staff* and fund an evaluation, how 
to support its proper implen^ntation, ^x\d how to insure that it is wcU- 
utilized (Hice it is completed. 

The chapter is present^ in several sections. This introductory section 
pie^ts some key concepts ^ut the nature of the evaluation planning 
process and the imp(»lant role played by tte evaluation platiner in that 
process. Tte central itteas expressed suggest a general framewoHc for 
planning the evaluation of social programs at both the state and local 
level. Later sections expand upon this framework, using the JTPA 
program as a continuing example. 

Effective planning for evaluation requires an expansive view of the 
planner's role ami the planning process. In this view, the focus of the 
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planning efToit is ncA narrowly coitered cm the research aspects of tte 
evaluation* Rather, anentitHi is time bromily focused on developing ami 
sustaining organizational support for the evaluaticm and utilizing its 
r^ts^ as well as rni implonenting itv^ evaluation. Hie planning pnx^ 
for the evaluation grows mit of important preliminary c^anizational 
work that sets the sta^ for later succet^ful implementation of tte 
evaluation, disseminaticMi of useful information, aikl utilization of the 
results. 

Hw ovCTarching {miKiple emphasize in this chaj^r is that of a user- 
fihiised approach to evaluation piannmg. ccmcept begins with the 
premise that if evaluation is to be truly useful in improving social 
programs, evaluation planning must start with the anticipated users of tl^ 
results arnl their particular informatics needs, and tten build from this 
essential base. Ideally, the user-focused planning appro^h consists of a 
series of sei^jcnced steps as CHitlined in chart 5.1 . 



aMflTS.1 
Shqis III fti Um^Fam84 Erakiitton 



• htenttfyir^ users and their mfmr^^ 

• Assessing the organbational supports to and a)nstr3int$ on the e^mlimtion process 
» De^h^^abeginr^»}{H)c ^twrnkfortfrnevaluatfc^. 

ItafVtopiBi i 1^^^ &nitairtN)i Strategy 

• itte* tifyfng vm< toy evaluation i^'^ and questions, 

• Merminingnwfea^imyof tlwqu^^. 

• Cfioosit^ a nmag^te set of res^nM)^ qt^shons. 

• S^ectbig an ^^.t)pfttm e^u^>n ap{m)s^ and ^bie m^hodofogy. 

PteM^ for to tat^B^i^ Uw Evatyattoa 

• Devek^riiq an imj^nenMrni pian 

• AssessinQ the re^Mifces needed and tt^rcrsts. 

• DevaiiH^a^ffirv^raiBgy. 

• AcQoift^ttw necessary res>uim 

liqilmiaft^ tti Cratavfhm 

• Cdiecting and ana^fzmg data. 

• tovek)]:^ «^iusmns and r«comnmtdatiof%, 

• D^senkmit^ the e^uat«»i infom^on to users. 

• tJtSizingtteevtfiiattonmfofniatk^ for (^skm making. 
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Hie chnMK>lo^ in chart 5,1 ^gi^te a rcat, logiod pn^rmscm of 
sieps« In i»wUi«, the chronology will not be so clear-cut. The process 
may require that multiple tasks be coordinated ami carried cmt within tte 
same time frame, rather than in a clearly ctefuied sequence. Ncmetheless^ 
cMitlining the icteal t^lps us think about and soA %nit tli^ lm>ad array of 
plaraxsng tasks diat will be need^ to suppcnt and sustain {m^gram 
evaluaticm. Tte seci»Ki section will expand upcm key compoi^nls within 
this fianKwork, using JTPA as the ca^ example. 

Tte integrating principle tying these tasks logelter is the assumption 
that tlK primary purpose of state and local program evaluation should be 
to im)vi(te inforauticm that ifrforms decisionmaking. It dKmld be 
relevant to tte u^rs, wrinen ard [Ktck^tged in a way that invites use, aiKi 
available Bt the most opportuiys time from the d^isionmaker^s perspec- 
tive. Otlwrwise, tte invesmient in evaluation may be wast^. The 
planning process must, therefore, begin with the u^r and his or her 
perceived mfcmnation ne^s, remain o^nizant of tt^se iKcds through- 
uut, and end with information that is actually put to use in improving a 
pix^ram. 

In the user-focused planning process, the role of tl^ evaluaticm 
[banner is distiiKt from tl^ fvaluator and centml to tl^ effort. In scrnie 
imtances, in smaller organizations, the person planning tl^ overall 
evaluation effort may also act as tte evaluator, i.e., tte of% who cksigns 
the evaluaticm, oversees its implementation, and analyzes and reports on 
Hxt findings. Nonetheless, the plani^r^s role in this prcx^^ is i^parate 
dsA distinguishable. 

Hie planner plays a key organizational role that womplements and 
supports the more technical research zKnivities of th^ evaluator. The 
planner acts as a crucial communicator and coonUnatm* among three 
major grou[^ involved in the evaluation {Kocess: iht evaluation research 
staff who must ctesign, implement, and analyze; the ^hninistrators and 
other ctecis](Hunakers who i^ed the evaluMion infcmnation; and the 
{aogram staff and clients who may be affected by ami, in timi, affect the 
evaluatiim jn'ocess and its outcomes. 

Evaluation does not oc^ur in a social vacuum. Just as ix)litical and 
raganizatimal factors influeiKC a prc^ram's design ami opeiBtion, so 
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will wch factors influence the nature, scope, and ultimate utility of an 
evaluation. A chief challenge for the planner will be to accurately assess 
and creatively bJild upon the organizational ctmtext in which the 
evaluation is to take place. 

At each stage of the process, tne planner will want to anticipate 
oiBanizational constraints lo and supports for evaluation and adjust 
planning activities accordingly. For example, in the initial stages the 
planner should focus on defining specific evaluation users ami uses. 
How might the organizational context influence the interest and partici- 
pation of potential users? Wiil inter- and intra-agency craiflicts hinder 
efforts to bring certain users together? Given differing oiganizational 
agendas, what kirals of ^ommodations to others and sui^rt fcM" the 
evaluation will different decisionmakers be willing to make? And. most 
important, what kind of benefits will users expect in return for their 
participation? 

The planner must also carefully assess organizational factors that may 
affect tl» actual evaluation process. How well the evaluator and 
evaluation activities are received by program staff, clients, and others 
involved in the process will depend largely cm the quality of advance 
oiganizational work umtertaken to prepare, educate, and appropriately 
involve these concerned parties. In the role of oiganizational communi- 
cator ami coordinator, the planner can help miticipate staff concerns 
about the evaluation and work collaboratively with the evaluator and his 
or her team to address these concerns. 

Frequently, a major organlzaticmal constraint to evaluation is lack of 
staffexpertise and other resources. In an em of scarce r^urces for social 
programs, the evaluation planner must unavoidably be preoccupied with 
resource acquisition and planning. What kimis of expertise will be 
requited, and how can that expertise best be obtained? What kinds of 
financial support and other n,.sourccs. such as computer services, will be 
necessary? 

In the evaluation planning process, the final principles to stress arc 
collaboration and partnership. Tlie importance of the planner's role in 
working closely with researchers, other staff, and evaluation users has 
already been emphasized. The planner is further challenged to go beyond 
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the immediate user circle to seek creative funding paitnerships in both 
tfie public aiKl private sectcns. The collaborative fuiKJing a}^ro»:h is an 
(HBsnizational investment that extols beyond the life of \hc individual 
evfUuaticm efToit. In seeking fumls outsicte the immediate program to be 
ev^uated, the evaluation pIiuuKr creates new networks of contacts and 
i»w possibilities for mitsifte community support and involvement in the 
program. Such support and involvement are likely to enbam% not only 
the evaluation efTort, but also tl» (ingoing program effort 

An amplification of the central themes presented above follows. The 
evaluaticm planning chronology, as cHJtlined earlier, serves as tl« general 
fiamewcnk for elaboradng cm the planrtng process and the evaluation 
planner's role in that pnx^ss. Within this framewoHc, ch^ter 
focuses on tlw first three ma^ planning stages leading to the ^ual 
implnnentation of ihe evaluaticm: 

1. Preliminary planning for user-centered evaluation. 

2. Developing a specific evaluation strategy. 

3. Planning for implementation of the evaluation. 

Within each of the major stages, the more important planning tasks 
will be highlighted. As in previous chapters, the JTPA program is used 
as a case example for illustrating key features of the process. 

Prelfminary Hannii^ fi»r a 
User-Centered Evaluation 

Even before specific evaluation questitms are cklii^ated or an evalu- 
ation appros^h settled upon, important preliminary planning issues must 
be consictered. This preliminary wcrk revolves around the following 
interrelated qi^stions concerning the orgmiizational setting in which the 
evaluation will occur 

1 . Who will be the chief users of the evaluation resuhs? 

2. What kinds of evaluation ^Krtivities are most feasible? 

3. How will ihe fiscal and organizational context influeiKe the evalu- 
ation effort? 

4. How can organi»/f lonal support for the evaluation best be developed? 
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The manner in which ttese questions are dealt with will have long- 
range consequences for the implemcniaiion of evaluation and its ulti- 
mate integrity as a useful planning, policy, and management tool within 
JTPA. This first section focuses on these preliminaiy planning coi»cms 
and the role of the evaluatior' planner in developing oi^aniatational 
support for evaluation. 

Defining the Users 

If evalualiCTi is to be pragmat^j. i.e., provide useful infonnation to 
decisionmakers for improving social programs, thai the evaluation 

planning pitxess must begin by aniicipaiingandictentifying the evaluation 
users and their iftformatian needs. These factors should drive tte initial 
fonnulation of the evaluation project and the planning steps that follow 
(PMton 1978: Davis and Salasin 1975). This kind of user-centered 
planning approach increases the likelihood that evaluation results will be 
useful and, in fact. used. 

A user-centered approach implies thai the evaluation planner should 
play an activist role in ittentifying. educating, and involving potential 
users in the evaluation process. In this preliminary planning sfags, for 
example, the planner has to target potential cusiwi^ and supponeis of 
evaluation and initiate cont^t, rather than wait for these parties to 
involve themselves. In collaboration v«th technical staff, the planner 
assists users in articulating their specific requirements, and suggests 
ways in which evaluatiwi might fulfill ihsse needs. 

The proactive planner needs to marker evaluation on several levels. In 
addition to inforroati(»i. evaluation offei? side benefits of which poten- 
tial users should be made aware. These benefits, such as improved 
agency coordination and cooperation ami increased political credibility, 
arc often intangible and not easily measured Evaluatirai can also be 
mariceted as a capacity-building investment in the (Hganization, yielding 
improved stoff capabilities for future evaluation or related research 
activities. It can lead to an enhanced Management Infomiation Systems 
(MIS) or other data collection system, and increased access to contact in 
research, professional, and private sector networks outside the organiza- 
tion's normal sphere of communication. 

31,: 
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Pb^ntial users of evalimtion need not be narrowly ctefim^* While 
intend and commitinent of pn^ram poltcymakcrs ai^ acfaninb^trators 
are key, c^her users^ both in and mit of die pn^rain* 2UK>uld ncH be 
overlooked. Additicmal candidates include planning and operation's 
stafTt whme input in shaping and refining tiK focus of ihc evoluaticm 
inquiry is valuable. Decisi(mmakers and staff from rclated progmms or 
agencies who might beiH^fit from tte evaluation rcsuhs should also be 
ccmsicten^* 

In the case of JITA, wtrcrc legislation mandates coordination between 
employment and training ind welfare ami educational agencies, ^^int 
support of evalimtion s^tivities is an import^t pmsibility to explore. 
Users frmn tlm^ coordinating ageiKies may have substantive contribu- 
tions to make in the form ol data, staff expertise, and political or fiscal 
support. Ttere can be a bcr eficial return on such contributions to these 
people in t)^ form of useful « ^va'uation findings, or iiKncased recognition 
and credibility as m evalupjcni i^ictpant* 

Finally, potential users may ii^lude a variety of gi ups outside JITA, 
such as elected officials and legisiaton^, clients and client-advcKjates, 
researchers, local business arnl labor groups, aini the genera] public. 
While nm all such users may be involved directly in the evaluation 
piocess, it is important to consider how ttif^ir interests and their informa- 
ticm i!eeds might affect tt^ ultimate focus of the evaluation. 

In icteniifying a range of potential users and their needs early on, the 
planner establishes a better position for gan^ring a bnmd base of 
cnganization^d support essential to the evaluation cfTort. How informa- 
tion i^eds ultimately translate into a specific nr^^'^arch design will be 
picked up in a later section* 

Determining Feasibility 

Before considering a specific re^arch plan, evaluation planters must 
study it^feasibility of evaluating a particular JTPA prc^ram. Aie some 
kinds of evaluation efforts more likely to suo:^ )han otters? !s the 
timing appropriate, or would an evaluation yield better results at later 
date? To answer these kinds of questions, a number of experts liavc 
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suggested that evaluation planners begin with an "evaluability assess- 
ment" of the program in question (Rutman 1980; Schmidt 1978). Such 
a piciiminary assessment, which may require assistance from an outside 
specialist, will help an organization accomplish the following goals: 

1 . Define the appropriate scope airi timing for an evaluation. 

2. Avoid wasting time and planning effort that will not produce useful 
results. 

3. Identify evaluation Iwiicrs thai msed to be removed before evalu- 
ation can take place. 

4. Lay the groundwork for (k>ing further evaluaticMi plaraiing when 
circumstances are more conducive to such efforts. 

Some of tlK mml obvious Iwriers to u^ful evaluations of JTPA 
programs are related to resource or technical constraints. In the following 
sections, some of the major implementaticMi issues coiKeming funding, 
staffing, and managing JTPA evaluation efforts will be jMcsented In 
greater tteiail. TTiese concerns aire Isiefly mention^ I»re as they tCHich 
on program evaluability. 

Financial constraiHs. Are sufficient funds available to ensure suc- 
cessful completion of the evaluation effon? If i,ot, can additional funds 
be obtained within an acceptable time frame? A scaled-down, l»it well- 
supported evaluation effort, providing quality infonnation in a few key 
arca-i. may prove to be the most useful interim option. 

Staffing! constraints, ln-hou.sc staffing of an cvaluatiim f ffort is one 
way to overcome financial constraints, but if staff resources arc thinly 
stretched, or if staff lacks the necessary technical expertise, this strategy 
may end up compromising iIk quality arni usefuIiKss of tte evaluatitm. 
An in-house evaluation may also lack sufficient credibility if the effort 
is perceived as jwlf-scrving. 

Evaluation timeframe. To be most useful, evaluation results must be 
available to users within a time frame that supports their decisionmaking 
needs. In JTPA, for example, evaluation activities should ideally mesh 
with the two-year program planning cycle to produce infcmnation for 
decisionmakers at key junctures within that cycle. 

Data collection proMems. Insufficient, inaccessible, or unreliable 
data may also limit the nature and scope of an evaluation effort. 
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Program Features AfTecting Evaluability 

Another net of factors affecting evaluability has to (k> with tte nm- 
tours of the program itself. A social program may exhibit certain char- 
acteristics that make evaluation (nitcomes mon^ difficult to interpret and 
utilize effectively. Typically, a process study may be necessary to 
elucidate such features before an organisation c^isicters evaluating. Tt^ 
process mode! presented in chapter 4 suggests key organizational 
components that might be useful in (tetermining pro^wn evaluability. 
For example, program gmls lune a central feature affecting evaluability. 
Explicit program gc^s provi(k^ a prettetermin^ standaxd against which 
program processes and accomplishments can be measured. When a 
program^s goals are unfocitsed or (x>nstantly changing, the task of 
evaluation is mone difficult. How do you measure your achievements if 
you are not sure about what you are trying to achieve? 

Alternatively, prognun goals may be well-defined, Init inconsistent 
with each other, complicating the task of evaluation* For example, the 
goal of achieving a high placement rate at a low cost per placement often 
conflicts with other goals, such as si^ificant participant skill dcvelq)- 
ment or long-term retention of traii^s in their placements. Such goal 
conflicts are interent to many JTPA programs. The issiK? is not to 
completely eliminate such conflicts, but to make \ht evaluation a}^nt>m:h 
as sensitive as possible to the ccMistraints placed on achieving program 
outcomes. 

The manner in which program services are delivered is anotter 
important consideration. When programs encom^ms numerous service 
IHt>visicm strategies (as is the case in many JITA i»T>gram settings), or 
change strategies midstream, the evaluation task b^omes more chal- 
lenging. The less uniform tl^ overall treatments provided, the more 
complicated the task of adequately accounting for program im{^ts* 

Finally, the size of tl^ prc^ram may shape the nature and M:ope of 
evaluation. In Ht^ case of smaller programs or pilot pro.^cts, impact 
findings may be of limited usefulness because sample size may be small 
or cost inefficiencies may exist. 
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Organizathnal Factors Affecting Evaluability 

OiBanizational filters often present the least tangible but most pow- 
erful barriers to useful evaluation. Some common or^anizatitmal factors 
affecting evaluability are su^sted below. 

Sti^ng proMems, Wten a program h plagued with low stall morale 
or high turnover, something is clearly wrong, Imt an evaluation may not 
I^lp. EvaluaticMi activities may create added burdens, which the staff 
cann(H handle. Effective employees arc crucial in any social service 
program. An organizatic»i with serious staff problems must focus on 
rectifying those problems before being able to utilize broad evaluaticm 
findings* 

Previous evaluation history. Have previous evaluations been con- 
ducted? If so, how have they be«n used? Have evaluation findings bc«n 
ignored or used to undermine certain facticHis or personnel within the 
oiganization? If so, the cr^ibility and usefulness of the new evaluation 
may be questio led and staff cooperation lost. Evaluation planners will 
have to develop initial strategies to build trunt and credibility. 

Hidden agendas. In some cases, the spon^ir of the evaluation is not 
truly ccmimined to an open imjuii^ into program operations from which 
the program can Jeam or improve. Instead, he or she may wmit to use 
evaluation to support a preconceived mAim about the program. 

Financial difficulties. When a program is struggling to stay afloat 
financially, Uie utility of mx evaluation is often severely curtailed. 
Administrative energy is necessarily focu^ on program survival rather 
than profeHuH improvement. TTie program may be able to take better 
advantage of evaluation findings when it is on a more stable financial 
foMing. 

Imer- and intra-organizational relations. Turf battles over clients, 
staff, and otter resources can comprcmiise the evaluaticm effort. If, for 
example, cooperative support among ageiKtes is lacking, tte evaluator 
may find ac<^ss to importMit sources of information blocked or delayed 
in ways that hurt the evaluaticm. A comprehensive evaluaticm planning 
effort will include strategics to ameliorate or compensate for difficult 
organizational relations. 
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An cvaluability assessment is not intended to discourage evaluation. 
Pan of the assessment task is to help progrsni t^raiors determine which 
fmor& can be manipulated to enhance overall evaluability. Once these 
are identified, the evaluation planning staff can actively work with 
program administration and other staff to create a prognun environment 
thai is more receptive to evaluation. 

Examining the Organizatfonal Context of Evaluatfon 

The JTPA organizational context is complex, cross-cutting all levels 
of government, and embracing numerous agencies and oiganizational 
agerelas. Because of this complexity, understanding how organizational 
factors might intervene to help or hinder evaluation is especially critical 
to the JTPA evaluation planning process. In addition, the eva* 
itself may subtly influence program processes and outcomes. The e. 
the context in which evaluation occurs and the manner in which evaiu- 
;irion is carried out interact to affect evaluation activities. For these 
reascMis, preliminary planning for evaluation must include a focus on 
how the oiganizational context will affect evaluation. 

The planner's challenge is to identify and work knowledgeably with 
organizatitMial constraints and supports to evaluation. Since these 
factors will vary from program to program, the intention here is to 
inovidc a general framework for imrorporating tn-ganizational issiws into 
the evaluation planning process. 

Organizational Inertia 

To accomplish their specified mi.ssions, oiganizaiions create mecha- 
nisms for promoting stability and efficiency. They develop structures 
that establish chains of authority and accountability, standardize opera- 
tions, and rouiinize and parcel out work in a specific manner. In creating 
stable structures, oiganizations also create vested inteirsts; a ma^ goal 
of the organization becomes self-preservatl(Mi. Over time, the very 
structures developed to enhance the organization's efficient functioning 
have a tenctency to become rigid and resistant to change. Change means 
more uncertainty, and as such, constitutes a threat to the organization and 
its vested interests (Weiss 1983). 
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The logic of evaluation, on the other haiwi, is based on the potential for 
v/utttge. Ideally, evaluation feedback offers a rational mechanism for 
planned change in the interest of prt^ram improvement. Therefore, as 
a harbinger of such change, the evaluatirai planiwr can expect to 
encounter some natural oi^ganizaticMial resistance to evaluation activities. 
Sometimes the resistance is not active, but takes the form of passive 
inability to mobilize for an evaluation effort. Sheer oi^anizaticmaJ 
inenia — tte urge to follow time-honored structures and patterns that 
have shaped the organization's identity — inhibits the evaluation under- 
taking. On the other hand, in an age of shrinking public rejsources, JTPA 
ami other programs are under cOTistam external pressure to improve their 
efficiency and effectiveness. Evaluation provides a tool for such 
improvement, which need not threaten tlK security and continuity of the 
organization. 

Overcoming organizational iiKJrtia or mitright resistance to evaluation 
may present a bigger challenge than the evaluati(»i itself. JTPA's 
complex administrative structure may demand that not one, but several 
separate organizational entities be mobilized to cooperate and participate 
in evaluation activities, if those activities are to be meaningful. A 
common organizational fear is thai tte evaluation results will point out 
«ily a program's weaknesses and tl m^e program credibility. I^x)gTam 
administrators and service {»Y)viders need to be assure that evaluation 
results can enhance program credibility in several ways. The fs^t that a 
program embraces evaluatitm as a tool for innovation and improvenient, 
itself, sends a positive message to program sponsors. Moreover, a 
balanced program evaluation will help identify program strengths as 
well as weaknesses, underscoring jHTogram accomplislmients that com- 
pliance measures alone may not reflet. Finally, evaluation may produce 
information that ccMnpensates for or explains lower compliance with the 
various performance standards required in JTPA. 

Orgemzational Roles and Rtlations 

In JTPA, numerous distinct state and local level organizations are 
involved in program activities. Often, at both levels, separate groups set 
policy, administer programs, and deliver services. In addition, clwrted 
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officials, busii^ss groups* other oiucalicHi or social welfare-related 
agencies, and economic development agencies may play an mrtive or 
influential role in JHTPA. AH itese actcm; have developed an org^iza- 
ticmal stake within tte JTPA system. But will they want to participate in 
and support an evaluation? 

A strategy for developing user particijHiticHi ami su^^rt in evaluation 
mu^ be inextricably ti^ to an examination and understanding of the 
broader OTganizationd context in which users are operating* l^refore* 
before approaching axKi involving various users, the planner needs to 
assess the roles tl'^se various organizational actors play within JTPA, 
How active or central a role does each organization play? 

Program administi^tive entities, for example, play such a key role in 
service delivery that in most instances their direct involvement in the 
evalualicHi planning will be critical. How rereptive to or ctmstrained by 
evaluaticm are key actors? What explicit or implicit agency agendas 
might affect the evaluation effort? For example, if an SDA h^ not met 
the fe<krally mandated performance measures, it may be interested in 
initiating its own evaluation but not interested in participating in an effort 
initial by others. Ignoring the interests of a jwticular JTPA stake- 
bolder in tl^ planning phase may impe<k tl^ evaluation in later implem- 
entaticMi and utilization phases. 

It is not sufRcient to know who the organizational ^tors are and what 
ttetr stakes in JTPA entail; must also know how ttese various groups 
interact with (me anotl^r. Do tt^ PIC, local program staff, local officials, 
and involved agencies regulariy communicate with each other? Are 
tl^re unresolved turf littles over JTPA or other program areas? Have 
personality conflicts mailed interagency cooperation in tlte p^t? These 
are the kinds of questions an evaluation planning group will have to pose 
and answer in onkr to lay the organizational grcnmdwork to suf^rt an 
evaluation effort. 

Sfmietimes organizational interests are pitted against each other in 
ways that make coordinated evaluation very difficult. Conflicting inter- 
ests £ue most likely to arise wl^re two ageinries share tl% same client 
base, as is the case with many JTPA and welfare programs. Competition 
between these two programs can be particularly intense when the fuller 
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funding of JTPA has translated into less fumiing for welfare. A welfaie 
agency will then perceive that it may ncH be in its interest to participate 
in an evaluatitm that might validaie JTPA at welfare's expense. 

If agencies have a history of poor communicatiiHi or struggle over who 
should administer what programs, or who should set policy, this histoiy 
can spill over into and stymie evaluation efforts in significant ways. 
Access to necessary data ot information on clients or pn^^s may be 
daiied or delayed, and otterwisc useful in-house resources 'nay not be 
discovered ami shar^. Moreover, tlw organizational input necessary for 
fcnmiulating ui^ful evaluation questi(»is may not occur, so that ' .< 
general utility of evaluati(m findings may be impaired or simply n..; 
rec(^nized by important decisionmakers. 

CfH^mive Phmniagfor Evaluution 

Conversely, identifying potenti^ly positive interagency connections 
provides a Isise on which to build the evaluation effort. Evaluation 
activities that cross agency or divisional boundaries, while providing 
exda challenges to planning and coordination, may also provide uniqi^ 
oj^rtunities for the exchange of information and iiteas within the 
overall JTPA organization. Since evaluation often r^uires special 
coordination among different units, tte process can create a suf^rtive 
context for interaction acruis territcmai lines. Such inier»;tion can itself 
be valuable in infmrning p^le about (tecisionmaking ara! vtoxk agen- 
das in different agencies, reducing organizational isolation, and improv> 
ing coordination of resources (Blalock 1988). Whatever the organiza- 
ticMial configuration, the planning role cannot remain (Mirely technical. 
TTie evaluation planner may neai to play information broker ai^ media- 
tor, siting as a conduit to open up or enlarge channels of communication 
and cooperation. 

Each stake-hol<ibr r^eds to gain something from partici{>ating in the 
ev^uaiion effort, whether it is information, public recogniticm. enhanced 
support, or credibility. A crucial task for the planitt' is to elicit from 
IMimary actors what it is they are willing to give and need to receive, in 
return, as i^rticipants in the evaluation process. The planner's task is 
also to help sensitize at»ors to each other's concerns, oringing covert 
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imtes into the bargaining arena (e.g., the peiennia] problem of data 
^uisiti»n across agencies) so that necessary agreements can be nego- 
tiated upfront before evaluatitni c(»ninences. 

Successfiil cooperative evaluation planning can be supported and 
sustained through a number of strategies, as summarized oelow, 
!. Inmhe key actors. Preliminary meetings with key actors in the 
evaluation pnx:ess will help shape an evaluation approach that ac- 
commodates a variety of comrems and does not exaceri^te inter- or 
intra-agency conflict Staff, as well as administrators, s^uld be 
included in early planning and/or briefing meetings. 

2. Develop advisory grtutps. An advisor> group is another way to hnng 
diverse organizational interests together in the evaluation {banning 
process. Group members can include not Ciily agency representa- 
tives, but outside professionals or otter citizens who can contribute 
expertise and lend additional support aral credibility to the endeavor. 

3. Develop mnovative funding and staffing alternatives. Sources of 
support for evaluation cTiist beyond the usual organizational chan- 
nels. Moving outside an agency for evaluaticm resources can extend 
the base of interest and support for such activity. 

4. Put interagency agreements and assurances in writing. Successful 
evaluation often (Spends upon interagency cooperation and re- 
source sharing. Since control of resources is always a sensitive 
(Hganizational issue, negotiated agreen^nts about wxes& to data, 
clients, staff, and other resources must be in writing to avoid future 
misunderstanding. 

5. Use a team planning approach. A team approach to planning makes 
sense when a great deal of interagency or intra-agency coordination 
and communication is necessary to accomplish evaluation tasks. 
Even if an outside evaluator is brought in to do the woiic a team 
might play a useful advisory role, providing a m^hanism for more 
direct oiganizational involvement and commitment to tte evalu- 
ation. In a JTPA evaluation, representatives from a vari.;ty of 
divisions or units within the overall coordinated system might 
contribute effectively to a team planning effort. Besides a member 
of the evaluation or research staff involved in conducting the 
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evaluation, team might also incorporate representatives from the 
program staiT(s^) involved, from relevant policymaking txxiies (such 
as a state or local coiUKril), mid from tite MIS or computer services 
division. 

Choosing Evaluation Staff 

The oiganizational context should also influence who pl^s, imple- 
meniK, ami administers a JTPA evaluation. Should t}^ employment and 
ttaining staff have primary respi^sibiiity for evaluation, or shcHiid a 
policymaking body like the PIC? Or, should an otganization more 
removed f^rnn the JTPA system have primary evaluation responsibili- 
ties? Should evaluation resj^sibtlities be divided anumg different 
entities? Clearly, given \ho enormous organizational variaticm across 
JTPA, no oi^ unit is tl^^right^ place to house an evaluation effort. What 
woii^ well in one setting may not transfer to anotlier. Some major 
considerations in choosing an evaluaticm staff include their position in 
the organization's authority structure, their objectivity, the degree of 
trust they engcncfcr and their specific research competence. 

AuAority Structure 

The positioning of an evaluation staff within an oi^anizational hierar- 
chy is important. Ideally, evaluaticm staff will be sufficiouly ctetacted 
from ^he existing hierarchy to hold no direct power over those being 
evaluate or, conversely, those in a program being evaluated do not have 
direct authority or influence over the evaluators. If the evaluator is 
thought to be too closely aligned with the administrative power structure, 
his her cr»iibility may be impai and with it tl^ ability to carry out 
evaluaticMi functions. On the othor hand, if the evaluator is perceived as 
lacking sufficiem administrative support, !^ or sl^ may be seen as 
"^marginal*" in relation to mgoing program operations. In this instance, 
the message being sent is that tt^ evaluation is not very important; 
co<^ration in tl^ "effort may again be uiKlermined. Greater staff 
detachment is ofien achieved by cc itr^ting out to a private consultant 
or establishing an independent evaluation unit. 
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Wlwn the heud of an evaluation unit reports directly to ma^ decision- 
makers in an oi^ani/aiitm, evaluaticm activities usually receive better 
suppoit fiscally ami politically, and evaluation information is better 
utilized by managers and policymakers. Such a direct link to power- 
hoklers, however, may have to be offset with cxtm effort to bring a range 
of apinopriate division sudministnitors and relevant staff into tlK plan- 
ning process. Chherwise. there is the danger that thoi% in lower echelons 
will feel compromised by or excluded from important (fecisionmaking 
and become less supportive of the evaluaticm effort 

The JTPA authority structure at both the state and local level is 
partially defined by who conducts compliaiKre-relatoi »:tivities. Mc^ 
JTPA organizati<ms have ck^veloped special m(»iitoring and compliaiKe 
units, which routinely collect and analyze JTPA program data ami audit 
certain aspects of JTPA program operations. Since these units are 
alre^y collecting information about JTPA, and since evaluation is often 
viewed as m elegant offshocH of mcmitoring, the temptation is to add 
evaluaticm activities to raigoing mcmitoring and compliance operations. 
This tendency is probably reinforced by tl« CETA legacy of minglmg 
ccmiplianc-* and technical functicms under one rcx)f. 

From a jniiely technical stamlpoint, piggybacking evaluation onto 
ongoing monitoring operaiicms may make sense: staff is familiar with 
tl» data, program oper^cms, and personnel. From an organizaiicmal 
standpoint, however, such an arrangement may be problematic. As 
mentioned ^lier, downplaying tl» threatening aspects of evaluation 
and enlisting the cooperation of those being evaluated are imporiiuit 
ingreciienls m planning a successful evaluation. The neutral, nonthreai- 
ening posture an evaluation staff s^ks is readily compromised in (he 
eyes of those being evaluated, if that same staff is also conn«:ted with 
a»npliance ^tivities. Tt^ inherently threatening aspects of evaluation 
are heightei^ by the fact thai the evaluating office is also the office that 
critiques ami sam;tions. A comfnomise appro^ might be to involve 
monitoring and compliance staffers as special evaluation consultants, 
who can provide unique information and insights into JTPA program 
operations, while others actually implement the evaluation. 
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in^pendenee and Objectivity 

Since the evaluation miK^sion h to yield the most accurate and c^jective 
information possible to decisicmmakers, iIk objectivity of those ccti- 
ducting the evaluaticm is a key coiKrem. An evaluation stafTs actual and 
pen: .ived m;utrality is closely connected to its position in Uie organiza- 
ticm structure ami hferarchy. The more involved or invested ;wticular 
staff is in the cmgoing planning, sdmini spring, ami implementing of a 
im>gnim, tl« m^re difftculi it is for it to carry out an objective asscsaneni. 
Its experiences with and prwonceivoi notions about the program may 
lead to tl^ uiKonscious filtering of what is observed axui how it is then 
analyzed, interpreted, and reported. If evaluator objectivity is qi^- 
tioml eitl^ by decisicHimakers or tlK>se being evaluated, the whole 
purpose of tl:^ evaluation effort may be called into question, and the 
potential utility of that effort lost. 

The quest for neutrality does not inevitably lead to expensive outside 
consultants. First, hiring outside consultants does not automatically 
remove the suspicion of bias — outside e valuators may merely be view^ 
as an extensic^i of those who hin^ them. Secoml, there are alternative 
appro£^hes to evaluation that sufficiently meet the requirements of 
independence and neutrality* For example, as mentioned earlier, evalu- 
aticMi can be acc ^mplishcd through an independent research unir, which 
is under an adminisUative authcnity separate from that of the program 
being evaluate. 

If an independent research staff is not feasible and a strictly in-house 
evaluation effon is contemplated, the evaluation planner must search for 
other structures or mechanisms to protect the objectivity and credibility 
of the evaluation. An organization might consider temporarily borrow- 
ing outsicfe staff or exchanging staff with ott^r divisions or closely 
related organizations, in order to achieve some greater detachment frmn 
the program on the part of the evaluator. 

TmH and Competence 

Tmst is anotter important consideration in deciding who is best able 
to carry out an evaluation effort. Trust enhances the evaluaior*s ability 
to gain entry to a program and elicit infoniiation and assistance from 
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program administration and slaJT. An evaluator^s neutrality does not 
necessarily guarantee trusn neither docs it necessarily cngcmter distrust. 
In fact, trust may be based on the evaluator's perceived positive bias 
towards a program. In selecting the evaluation staff, tradeoffs may have 
to be made between the researcher who has greatest rapport and access 
to iffogram information and the one who exhibits the greatest neutrality 
and incfepoidence. 

The technical competence of an evaluation slafT is a primaiy factor in 
deciding how best to build an evaluation capability. If the technical 
expertise is inadequate or inappropriate, an evaluation is more likely to 
waste resources and pnxluce results of questionable validity and useful- 
ness. However, technical competency and efficiency, while of primaiy 
importance, should not be the sole criterion for location of an evaluation 
effort. Familiarity with JTPA programs and the ability to maneuver 
within that system to accomplish goals are also important attributes for 
an evaluation staff. 

The more comprehensive the evaluation effort, the greater the need to 
involve different constituencies and coordinate their activities. Who is 
best able to perform vital coordination efforts, to bring interested jwrlies 
together in critical planning stages, to establish interagency agreements 
about data and resource sharing, to bridge communication gaps when 
necessaiy? Here again, some aigue that these critical nontechnical 
competencies must be obtained by hiring an outside consultJint, whose 
vision can transcend the narrower perspectives of individual JTPA 
personnel. On the other hand, in-house staff, by virtue of its superior 
knowledge of interagency history and personnel, may also be in a good 

Preparing and Involving Staff 

Even if only temporarily, the evaluator becomes a pan of the twganiza- 
tional landsc^ in which he or she is operating. How tlK>se being 
evaluated peireiw the evaluator and how the evaluator, in turn, interac ts 
with these he or she smdics inescapably influenc-es the evaluation process. 
Therefore, the evaluator must be sensitive to the role as innovattM- within 
the oi^ganization and anticipate potential difRcultics arising from that 
position (Rodman and Kolodny 1977). The firet challenge for the 
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positimi to peifOTm such $ui^rt*iniilding ami coonlinaUon functions* 

McM^ evaluators regard themselves us facilitators of [Kisitivc change. 
However, it is difficult for tlK^ being evaluated to emlKm:e this posit ve 
point of view; they assume that the eva^luator has corr^ to point a 
dis^IHOving finger at what ihey are doing wnmg. If nothing is doi»s to 
soften this i^ative view of the evaluator^ that is, if no assurarKre and 
piotecticm are givra to tte evaluated, tl^n an evaluator^s piesemre is 
likely to induce a defensive posture that is not conducive to lite uhimate 
goals of tl^ evaluaticm. 

If progi^ staffers fc^I unsure of th^ purposes behind xiw evaluaticm^ 
their (tefcnsive ^ions can seric^siy uiutermiiK ttv^ pioce^ For 
example, in one case, JTPA evaluators were investigating tte imi^tsof 
a spixial JTTA program thrmigh use of a control group of nonfMurtici- 
pants Wl^ the evaluation was in progress, tte evaluators discovered 
that program staff members, in their eagerness to prove the program^s 
worth, hi!d become imoffscial jnogram gatekeepers, assigning JTPA 
services only to the most obviously ^b-ready. As a result, evaluators had 
diniculty assessing wl^tlKr positive mitcomes were due to the program 
services to die select nature of clients reiving those services, 

Tl^^se organizational diffi/ulties can be minimized if the evaluation 
planner devotes a sufficient amount of j^liminary plaiining time to 
appn^ately involving prc^^am staff in the process. As potential users 
of the evaluation, several staffers might particij^te in tlu; initial evalu- 
ation planning oradviswy group. Later, all afforted program staff should 
have an i^>poTtunity to meet with tl^ evaluation staff for a Mefmg on the 
planned evaluation i^tivities and purposes. 

Unavoidably, the evaluator has an effect not only on tte social climate 
of a program (an intructer on sxred soil), hit also cm tlv working 
conditions w ithin the program* In requiring interviews and meetings, tl^ 
evaluator distracts staff and ^ministrators from tl^ir regular work \imd. 
Wl^ther staff numbers perceive evaluation duties as a burden or an 
intrusion cteperKls, in part, on tte sensitivity of the evaluator asKi how well 
staffers are briered as to tte nature of the evaluati<Mi aiKi the importance 
of their role in the process In a positive ccmtext, evaluaticm interviews 
and planning mattings can oHer program staff a chance to be l^ard and 
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make a meaningful crnitribution. 

Eva'uaticHi planning staff can gmooth the way for tl^ evaluator by 
ediH:ating others involved aiMl woiicing out a clear (felii^icHi of every- 
ore's role in the evaluation, including the degiee of staff participation 
aad staff responsibilities related to the evaluation. Establishing infomuU 
chamois of communicatlcMi between the evaluator and others involve 
in the evaluation process will help reduce inevitable tensions and 
miscommunication. and protect the ongoing procedure. 

Finally, the evaluator must confront the possibility that his or l^r 
}»eseiKe ctMistitutes an sdditional influence affecting the program in an 
unknown fashion. If, for example, the evaluator is seen as threatening, 
staff morale and program effectiveness may decliiK. On the otter hand, 
because of the evaluator's presence, staff may lake extraordir^ meas- 
ures that artificially and temporarily boost program performance. 

Even if the evaluator is v»cwed in a strictly neutral light by staff, the 
subjects of the evaluation — who may range from JTPA clients to PIC 
members — may react to the process of being studied (the well-known 
"Hawthorne effect"). As a result of being observed or interviewed, 
subjects may conscimisly or unconsciously alter their behavior, biasing 
the evaluati(Hi results obtained. While such influeiKes cannot be totally 
eliminated, the plamwr can telp .sensitize the evaluator to the organiza- 
tioml setting in an effort to minimize bias in the evaluation process and 
its results. 

Reducing the Threat of Evaluation 

TTie evaluator is not automatically doomed to alien status within a 
hostile and mistrustful program environment. Although some organiza- 
tionsd fact(M^ may be beymid his or her control, tlw evaluaticm planner 
can ctevelop strategies to demystify the process and reduce a program 
staffs initial fears. Such strategies can include the following: 

1 . Involve program administrators and program staff in initial and sub- 
i^i^it evaluation planning activities, in onfer to enhance user 
understanding and ctMnmitmen* to the evaluati<Mi. 

2. Make clear to program persomwl the purposes and antidpated con- 
sequences of the evaluation. IdesAly, consequences center around 
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axiKUwrti ve program change, giving program operators room to ex- 
perimoit, learn from mistakeg, ai^ improve fm^grams. 

3. Emphasize the nxEhmtiim (]f programs, not personnel. Hie more em- 
phasis pl^:ed on evaluating program attributeH, as opjx^^ to ^afT 
attributes, the less threatening the evaluation process. If staff inatte- 
quacics are a iH^termii^ central a>n€em, ^en otl^r vehicles, 
such as in-service training, ^ould be considered to address this 
problem. 

4. Establish clear lines of authority separating evaluation staff from 
pn^ram administration staff, 

5. Introduce an initial evaluation effort into the least threatening 
program situation. For example, fcKUs initial inquiry on overall 
program structures, processes, or outcmnes, rather than on individ- 
ual service providers. 

6. Assure confidentiality to clients, staff. aiKl all other participants in 
evaluation. 

7. Select evaluators whose organi/ationai status is perceived as most 
neutral and nonlhreaiening. 

Developing a Speclfk: Evaluation Strategy 

Tlw previous section began with a !^t of key planning qti^stions abcnat 
evaluation usetii. program evaluability , and the organizational context in 
which evaluation occurs. We turn to an additic^al set of questions 
associated with developing an evaluation research plan, which will le^ 
to a specific e\xiluation strategy. 

1 . What ar? tl^ im{K>rtant questions that users wish an evaluation to 
answer? 

2. What general evaluation approach is moai lea^ible for answering 
such questions? 

3. What data will be required, and what demands on tte organizaticm 
will be maik in terms of data collection and ^alysis? 

In discussing issues raised by each of these questions, it is important 
to distingui^ betwwn research tasks ami evaluation-planning tasks. 
AltlM>ugh these may be performed by the same smiividual(s) in small 
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JTPA oiganizaticms, commonly some or all arc performed by various 
tr»iividuals with different expertise and different ^itions vis^-vis tl^ 
JTPA organization. 

The re.«^arch steps involved in developing a specific evaluation 
research strategy, as discussed in chapter 1, requite imiividuals with 
ai^propriate research expertise in order lo ensure the technical compe- 
tence of the research design and its implementation. Hie research tasks 
performed by tt^se experts, however, are o^mplemented and supported 
by a series of organizational planning tasks, as illustrated in chart 5.2. 
The evaluation plwi^r worics in partnership with tte researcter/evalu- 
ator to ensure user involvement in and general oi^anizational support for 
the development of a woiicable research strategy. 
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Identifying Questions 



As stressed in the previous section, an evaluation^ usefulness hinges 
in large measure on providing information that users r^ed in order to 
make more informed decisions about their programs. Hie actual design 
of an evaluation* t^Njrefore, develops around a key set of research 
questions about JTPA' s effecti\*eness and efficiency^ which flow directly 
from users^ interests. These key questions will, of course, vary at 
different points in time across different state and local pn^om settings. 
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but in general, useful evaluation will concern one or more of ihc 
following gei^ric questions: 

1 . Was the program implemented as planm^? 

2. Did the pn^riun thieve its stated goals? 

3. Did progTEun participants as a whole benefit significantly? 

4. Who benefited ^-nost/least fhrni the program? 

5. Did the program have unintended results (good or bad)? 

6. Which program activities were most/least cost-effective? 

7. How might implementation be improved? 

Defining the most significant questions about JTPA will help set the 
parameters of an evaluation effort early on in tl% planning process- 
In this question-fOTmulaticHi st^e, tte evaluation planner plays a key 
coUaborative lole with users in shaping tl^ direction of the evaluation 
effort. As mentioned at the outset, their participation is cnicial in the 
evaluation planning jntxress. User input irKrreases user commitment to 
the evaluation effort and focuses that effort on relevant issues. During 
tlK question-formulation stage, however, evaluation staff do not have to 
defer exclusively to users. 

Identifying specific questions can be a difficult process, and users may 
have i^vblems ^veloping researchable iix^uiries about the program. 
Because JTPA is so tremendously "performance driven," users may have 
difficulty moving from a compliaiKe and monitoring mode to bnmder 
inquiries. In such ca.scs, the evaluation planner plays an important 
educative role in eliciting or reformulating questions. 

Different Users, Different Questions 

Bringing different users into the question-formulation stage creates 
additional challenges for the evaluation planner because different users 
may be interested in entirely different questions. For example, ccmflicts 
may surikre between different decisionmakmg levels or iH'aiKhes of tte 
program as to what is truly important to know abmit JTPA. At the service 
delivery level, program staff may be more interested in the JTPA 
intervention's impact on clients. Are clients being placed effectively? 
PIC members may be more concerned with the business community's 
perceptions and mvolvement; administrative usen> may be more in- 
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trigued with studying the cost-«fTectiveness of JTPA. Political leaders 
may look for informatioi that justifies public expend«tures or responds 
to coimituents* needs. 

When state ami SDA users are jointly involved in evaluation, thorny 
i^ues regarding the focus of the evaluation may arise. Since the state can 
ultimately saiK^ion a poorly performing SDA. that SDA must be moie 
directly and unyieldingly concerned with performance issues. Stale 
JTPA policymaker^ on the other hand, may feel le^ compelled to 
examine immediate performance outcomes, ai«i focus instead on more 
Icmg-term effectively^ measures of the program. 

The question-formulati(m stage can provick: an ^kliticmal opportunity 
for infcmnation exchange and acc(»nmodation between these different 
facticKis. Part of the planner's job, then, is to stimulate this exchange and 
assist in identifying those questimis for which tliere is Glared interest or 
gci^ral support. 

D^rmMng PrUmty Questums and Their Feasihility 

Once users ami planning stalf have generated a number of evaluaticm 
qiKStions, tiny must be prioritized and tl^ scope of tl^ evaluation 
detenniml, l»sed on allotted time and resources. Though they s^m 
impcMtant, s(»ne questions may have to be eliminated because pursuing 
the answers will prove too time-c^suming or costly. 

Attempting to answer too many qi^ticms in raie evaluation efftnl is 
a common pitfall. When tte sc4^ is too omiiose. staff and otl^ 
resmuces may be too overextended to produce a quality (mxiuct. An 
overly ambitious scope imneas^ xht complexity of tte evaluation 
process and tl:«: coordination of staff ^tivity, and increases d^ likeli- 
hood that deadliiKS will be miss^ ami budgets overrun. In «idition, 
some qi^stions simply may m)t be fea^ble to study ai a }»iticulartime, 
given the existing resources available. TTierefwe, (kfming priority 
qi^tions as eariy as possible creates an important foundation for later 
evaluation activity. 

Prioritizing a set of evaluation issues can be «ie of the more fnistrat- 
ing and time-c€Nisuming stef» in planning a JTPA program evaluati<»i. 
Tte planner may have to sustain the process with a generous Aosc of 
i^otiation amcmg different users. In onter to arrive at an ultnnate \\A 

^ 



326 Evaluating Social Problems 



of shared priorities, the planiKr may have to sketch several different 
evaluation scenari(» and ^companying contingencies concerning fund- 
ing, staffing, data collecticm. and analysis. Users may then be better able 
to revise their qi^tions and agree upon priorities. 

Those engaged in setting priorities must have access to expert research 
advice as to the feasibility of answering each preliminary question from 
a research standpoint. If staff research expertise is limited, this is tix time 
to Ining in an outside research consultant to help seim and refine the 
questions. This consulting time is well-spent if it yields a manageable set 
of questions that reflect users* evaluation priorities. These questitMis will 
form the heart of the evaluation and inform and direct tl*e research efforts 
that follow. 

Sdecting an Evaluation Approach and MethodoH^ 

Once key evaluation questions have been selected, they must be 
translated into tte research inquiries to be ackiressed by the evaluaticm, 
i.e., the specific program variables of interest and the critical relati(Hi- 
^ips among these variables to be studied. A sul^equent task is to choc^e 
a research strategy to use in studying th^se relationships. The issue at this 
Sage is to select the most a]^)ropriate research approach, given tlw nature 
of the questions and the status of resources such as time, staff expertise, 
and data accessibility. 

This book illustrates three major kinds of program evaluation: net 
impact, gross imi^ct, and process evaluation. Each ^proach has its 
strengths and weaknesses and is mast apfM^riale to answering particular 
questions, as described in chapter 1. A comprehensive evaluaticm 
a^Mmrh combining both process ami mv^t questions is ideal; it yields 
useful information m program outcomes and on the pivcesses that have 
OMitributed to those outcomes. 

A number of factws in the real world will influence the kind of 
evaluation approach selected: evaluation costs, time frame for acccrni- 
pHshing the evaluation, data requirements, staff and oti^r resource 
cap^lities, and organizational demands. The evaluation planner must 
work in concert with program and research staff to ad^uately assess 
these limitations. 



ERIC 



User-Centered Evaluatkm Planning 327 

Settling on a basic evalu^on aj^roach is the beginning step in a series 
of research and planning decisions regaiding tmplen^taticm of the 
evaluadtm. In devel(^ing a feasible research design, staff must first 
def^ds who will be studied and how tl^ iw^ssary data will be collected 
and analyze. (The specific tasks involved in develqping a (tesign are 
(Ascribed in chap^ I and illustrated in chapters 2 through 4 in this 
volume.) At this point in the pnx^s, the evaluation planner plays an 
important organizati(»ial su|^XMt role for the technical evaluation de- 
signers, ensuring that the technical requirements of evaluation 
design m^h with oiganizational cajfflbilities. 

Here again, real worid coi»i(krations impinge upon the choices 
evaiuati(»i (tesigneiB would icteally make. Tl» full range of data desired 
may be too costly or tlme-C(m$uming to coll«:t. Sonw information may 
be difficult tc retrieve or merge with otlwr data sets. Staff may lack 
expertise in specific kinds of statistical analysis required by a research 
approach. Working closely with research staff, the planner's task is to 
identify resources and information r^uirements suggests! by a prelimi- 
nary research plan, and anticipate the various organizational factors that 
may enhance or ccmstrain particular design options. 

Anticipathig Data Coilectbn Issues 

At this stage, data collection may pose special issues for d» planiwr 
and the researcher to acklress. Wlwther data are derived from an MIS or 
otl^ aut(»nated data l^se systems, access to accurate and valid informa- 
tion is key to designing aiKi implementing any evaluation. Wittout 
iKleqimte data, most carefiil design may be worthless. Plani^rs 
should not wait until the evaluation is in progress to study data ga^ring 
systems, and then discover their inadequacies. Rather, these systems 
should be expior»i ami their drawl^ks uncovered in ti» early evalu- 
ad(Mi planning stages. The planner can directly support the researcher 
in this exploration in a number of ways. 

Data ReHabiiUy 

One of the researcher's primary concerns is the quality of the data. In 
part, data quality is a function of the reliability of the data gatl^ring 
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psrocess, that Ls, how aanirately and consistently the data are collected. 
For example, in an evaluaticni using MIS inforniaticm there are several 
potential sources of unreliable data: ( 1 } the client himself or h^self, (2) 
tte staff receding information s}xmt the client. (3) (t» data entry staff 
transferring that information, and (4) the system classification schemes, 
yMch do not cleariy or consistently distinguiiili one data element 
category from anotl^r. 

SDAs with highly dec^ntmlized intake and service (tehvery systems, 
where many different personnel in different ageiKies input data, have a 
grei^r pcHential for data inconsistencies and inaccuracies. The planner 
can assist the researcher either by directly reviewing the organization's 
data collection procedures and safeguards, or by making the organiza- 
tional contacts necessary for the researcher to undertake this review. 

Data ComptmibUity 

CotiVj^son of evaluati(m data across different subunits, such as 
states, SDAs, or even service providers, may be anotl^r ctsicem for the 
researcher. In onkr to evalual^ jm^ram implementatitm or outc<»nes, 
definilions of data elements across systems must '>e reasonably standard- 
ized. Achieving such staiKiantizaticm across different JTPA jurisdic- 
tions is often complicated, esprciaily in states that (^rate a more 
decentralized MIS system. 

WIktc JTPA sm^ices are ctemitralis^ amoig numerous separate 
contractors, the issue of data comi»rability extends all tlK way to the 
service-provi(k;r level. When the SDA or a proxy agemry, like the 
Employment Service, performs centralized intidce aiKi service assign- 
n^t, it can exert more control over tl» way in which {»rtici}»nt 
information is categorized ami codified in the MIS. Wtere tl^se initial 
service functicms are reliiKjuished to independent ctmtr^ors, however, 
standardizatioi of information is more difficult to maintain. Rig(»t>us 
categmzing and coding guidelines for c(»itnictors may iK>t exist; or, if 
they do exist, they may be hard to enforce at the subcontractor level. 

Again, the planner can play a support role by gathering information 
from the various organizations concerned so that the research staff can 
better determine wl^ther a data companibility problem exists. If 
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difTcrent units within ihe system are measuring the same vari^les in an 
inccmsistent manner, ti^; piamier will have to assess the organizational 
feasibility of Imnging greater uniformity of measurement systemwicfe. 

DataAveilabUity 

In any state or local setting, the Management Information System 
(NfIS), which provi(ks ongoing informaticm on a number of importam 
client and {v^ram vai.ibles, will be a key factor in the evaluation. 
Besides data quality and comparability, a primary concern must be ihc 
ability of the MIS to meet imp(»tant data m^uirements of evaluation. 
What (temands, in fact, will evaluation place on the MIS? If the MIS 
lacks certain data elements useful to evaluaticxi. how imUly can the 
system be revised? It may be more cost-effective in tte long run to 
hammer out a thorough revision ba^ on multiple evaluation uses, rather 
than slowly to attack a system pi^meal. 

The cost of adding elements to the MIS is an obvious constraint to 
modifyLig the system. In the more decentralized state settings, where 
SDAs operate in(k;pendent mainframe or software systems, irulividual 
mod'Hc^ti m may be especially costly because the states are likely to 
bear less responsibility for loc^y run inf<nmati<Mi systems. 

Computer programming time is not the rally cost issue involved in 
quiring r^w data for evaluation. SDAs n^ to be sensitive to the 
pcMsntial burctens (designing new forms, training intake perscmi^l, etc.) 
that adikd reporting requirements will place on themselves and tl^u* 
service provitters. Also, tt^re is a limit to tiK amount of research 
infcmnation an SDA or service provider can collem without compromis- 
ing its social service mission. TlKirefOTe, ]wt of initial evaluation 
planning must involve the integration of an evaluati(m*s MIS r^uire- 
ments into the SDA*s overall information needs. 

In the more centralized, state MIS systems, an SDA will have less 
latitucte in inctepoKlattly modifying its MIS. Umger-r»i^ planning f&c 
evaluatirai ^vitie<: likely must entail bringing t<^ther state and SDA 
u^rs to develop dn MIS capalnlity cniented toward bc^ parties' evalu- 
ation m^eds. SDas may have differmt inform^on priorities from each 
otber ami from tlie statt. however, ccnnplicating the task of enhancing t)» 
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MIS to meet diverse evaluatitm needs. In some instam:es, SDAs have 
collectively negotiated changes in propi^cd statewide evaluation to 
iiKlucte gattering more infonnalion of direct ccHwem to the SDAs. 

Data Merging 

Although MIS infomiation will often be at the core of many JTPA 
evjduations, additional inibimation may also be critical. For example, 
n^png MIS client data with oti^ kimls of client <^ta on post-JTPA 
earnings, employment, and welfare depenctency permits a more sqihis- 
tic^ed analysis of program outcomes and impacts. Frequently these 
ffilditicHial data are contain^ in data l^se systems cc»npletely separate 
and incompatible with JTPA MIS. The evaluation plan should anticipate 
the technical difficulties in bringing various data systems together for a 
unitary analysis. 

Tht task of merging MIS with other data involves organizational 
considerations as well. The data may be uncter another agency's 
audiority, and obt^ning that data may pose additional problems. 
Commcmly. data r^uests across agency boundaries are viewed as an 
imposition, requiring extra staff time or other resounds. Morrovcr, the 
outside agency may be under a different jurisdictirai than the JTPA 
agency (state rather than local, or vice versa). There may, in this case, 
be less organizational precedence or su]^xh1 for interaction aiK} coopera- 
tion with the JTPA agency. 

Sw^h realities underscore tte need for strategic organizati(Hial plan- 
ning as part of tlK overall evaluation planning effort. Repre^ntatives of 
affected ageiKries diouki be brought into the planning proc^ early to 
ext&ure greater cooperation. Interagency understanding about data shar- 
ing and computer use should be put in writing as further insurance against 
future frustrations or misunderstandings. 

C^nt ConfidenttaUty 

AltlK>ugh state ageiKies and SDAs may routinely ^lare JTPA client 
information, client confickntiality does become an issue as long as 
such information is presented in the aggregate without iiKlividual iden- 
tifiers, such as client name or social security number. However, both the 
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and grms imi^Kn evaluation approactes ctef^cribed in this volume 
involve merging MIS data with other data sources for which client 
tdentifkFS are require to acccmplish an information match. State 
statutes on client ccmfickntiality may be restrictive regarding the release 
of this information to otlK^rs. Some SDAs have encountered difHculties, 
for example, in obtaining stale-administerKi UI Wage Records. 

When two or more sepsu^te ageiKies agree to share JTPA client data^ 
issues of cimfickmtiality must be understood. E^h ^ncy may have its 
own internal stand^s regarding access to arid use of client data For 
example, or^ agency may strictly limit information containing client 
idoitiflers to a small number of sp^ial users, while others may allow 
wide access to such information. Some agencies may permit client data 
to be used for compliance investigation, and otln^rs may not. 

Assurances about confidentiality are espwially important to service 
providers because inability to guarantee client confitkntiality will impair 
tite client-service-provider relationship and sut^^equently affect treat- 
ment success. For these reasons, interagency discussion and agreement 
about client confidentiality must be part of the evaluation planning effort. 

Planning for the Implementation of an Evaluation 

Once staff has (teveloped a feasible evaluation approach, planners can 
think nK)re specifically about how the evaluation will be implemented^ 
and chart a course for the planning activities that implementation 
requires. These activities will center on assessing the resources nmled, 
estimating their costs, and developing strategies for acquiring and 
efficiently allocating them. 

Developing an Imi^mentatiim Kan 

A written implementation plan is an invaluable tool for conceptualiz- 
ing and carrying out well-cooixiinated, timely, and useful evaluation 
activities. Ideally, ^h a plan comprehensively documents all planning 
and management d^isicms that must [recede ^tual implen^tation of 
the evaluation* This plan is, therefore, an iixlispensable companion piece 
to tl^ research plan. The research plan documents tl^ specific research 
tasks to be unctertaken; the implementation plan details the organiza- 
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tional resources and activities that support the research. 

Committing this plan to writing helps in several ways, Firet, a written 
plan creates a conceptual record to which cme can c<mtinually refer for 
clarification and dir^ion. As a written record, the plan is more subj^ 
to outside review, critique, and revision. A wrinen record also allows for 
a more broadly shared understanding of the evaluation pocess axtd the 
planner's conceptual work that shapes that process. Such an understand- 
ing is crucial to the evaluation team for efTicirait coordination of tasks, 
particulariy between the technical staff and others involved in the 
evaluaiicm process. Evaluation users will also appreciate the opportunity 
to review the complex CMgani;rati(»ial considerations that contribute to a 
sound evaluation plan. 

More than a single document, a compretensive implementation plan 
ccHisists of a number of interrelated statements, descriptions, charts, and 
checklists. Infcmnal nmes, memos, and interviews serve as supporting 
or supplemental documents to the main plan. A plan should ctMiiain a 
purpose and goal statement as well as users' questions to be ackhessed. 
A puTfKJse and goal statement is the organizing principle behind both the 
research and the implementaticm plan. At tlte end of the evaluaticm, this 
statement offers a yardstick for measuring ti^ evaliuition's accomplish- 
ments. Was the evaluation implemented in a manner consistent with the 
mginal goals? How well did it answer the questions originally posed? 

Whatever written format is used, the core of tte plan should provide 
a detailed blueprint of the sequential activities occurring in each phase of 
the evaluation. The evaluation process encompasses three major phases: 
& planning phase, an implementation phase, and a reporting and dissemi- 
nation plutse. T)xs implementation plan ^i^ntially cmters all tl^ antici- 
pated evaluation-related activities within each phase, highlighting how 
cHganizational planning tasks dovetail with and supfxnt research tasks, 
as illustrate! in chart 5.3. 

In serving as the evaluation's blueprint, the core of the implementation 
plan covers activities as well as the timing, management, resources, and 
costs that these activhies entail (Adtjns and Walker 1979; Fink and 
Kosea}fT 1%2). E»:h of these elements is consider^l below. 
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Tht Time Schedule for Evaluation 

As with any work plan, the evaJuaticm implementation plan 

diould also imrlude a sp^iHc scl^ule for accomplishing tasks. Thi.s 
scl^uling dimension is important to the evaluation effort for reasons 
that extend beywid efficient day-to-day management and resource 
utilization. If not accomplished within a specified time frame, evaluaticm 
results bTCOTie stale. Tte organizational momentum behind the effort 
may die ami the results, when finally pnxiuced. may no longer be valued 
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or utilt^:^. Over time, the potential users of t}K evaluation may change 
substantially, and ikw users have less rommilmcnt to or inteiest in ite 
evaluation or feel more ikreatencd by the information tl^ evaluaticm 
elicits. For these reasons, user input should inforai the st hedulinf!, as 
well as the tvntem of the evaluation. 

jScheduling evaluation activities ideally should mesh with relevant 
fuiuiing. legislative^ and plaraiing timetables. For example, evaluation 
findings with implications for broad policymaking might ideally be 
coordinated with the policy time frames of tte PIC, economic develop- 
ment agencies, or local government, JTPA evaluaiicMi plaiming might 
also be coordinated with allocation decisicms for state set-asiite monies 
or other state arni local ^ministralive ^tions. The impOTtant point in 
overall scl^uling is to seize coordination of^rtunities with ott^r 
actors within the total JTPA system whenever possible. Such coordina- 
ticHi can cmly enhance the ultimate utility of the evaluation effort. 

Monitoring the Plan 

In scheduling evaluation activities, planners can build into the evalu- 
atiai planning pnxrcss opportunities for review, comment, aiKi revision. 
Opportunities to monitor e£K:h significant evaluatiuu pliase can mharK:e 
the overall evaluation effort in sc vr inl ways. Monitoring buil(k flexibil' 
ity into the implementation plan, allowing for changes and imjwovc- 
ments wheie necessary. Monitonng also encour^es the timely discov- 
ery ami conection of research problems or planning gaps, ultimately 
saving time and resources. Finally, external review by an independent 
third party can inci^ase usei^* vonfidem e in a predo^ninantly in-house 
evaluation and its overall credibility. Wl^n an evaluation eflbn cross- 
cuts organizational divisions or agencies, review takes on an added 
din^nsicMi. Wlwther formal or informal, Uie ongoing review process, by 
inviting feedl^k, can be an effective mech^ism for sustaining interest 
and involvement by initial sup|K>rters and {wtici{^ts. 

Resources Required 

As evaluation ne^s^ interests, and capabilities vary acrc^ local 
setting, so will the required implementation resources. As part of ite 
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overall plan, a written strmegy, or resource plan, for acquiring appropri- 
ate capital is essential to planning and managing ihc evaluation effort. It 
may b^in as a toitative docunv^nt for debsue ami revision in ti^ initial 
sta^. Before tte mmial evaluation focus hich questicms arc to be 
answered?) and aq^proach (what evaluation design is appropriate?) are 
(felii^oi, tte plan will be sketchy, but as eariy decision points arc 
rej^ilKKi, die plan will take on greater detail and form. 

A resource plan can be devised according to a number of formats. 
Whaf-ver formal s chosen, the basic elements of the plan include the 
following: 

1 . A sequential listing of evaluation tasks to be performed and products 
to be produced. 

2. A time allotment for each task. 

3. TTie staff and otlwr resources needed for each task. 

4. An estimate of the quratity or amount of resources required (number 
of staff hours, con , uiation or word processing time, etc.). 

These elements must be identified in writinfi and combined in an easily 
reliable form. Chan 5.4 contains a sample formal of an evaluation 
resource plan. As this design suggests, program evaluation will often 
r»}mre special staff or consultant in^t at key junctures. 

Deterinining Costs 

Estimating evaluation costs is a critical step in the planning prcnress. 
FUndcrs must have prelimmaiy cost parameters before authorizing an 
evaluati(m effort, and evaluation planners will want to anchor evaluation 
options to fmancial realties as early as possible. The thorough costing- 
oui of the major evaluation compofients, as listed in a preliminary 
resource plan, provi^s a practical basis for comimring evaluation 
alternatives and assessing the relative merits of different data collection 
and staffing strategies. An estimate encourages plani»rs to rethink 
alternative restnirce and staffing strategies more creatively, or consider 
one or more scaled-down versions of the preliminary evaluation design. 

Costs will vary tremendously depending cm the purpose and scale of 
the evaluation effort, the kinds of resources an oi^pnization can marshal 
to uiKtertake the task, aiKi the existing market cost for external resources. 
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such as consultants. For example, coasuliant fees for an evaluation 
specialist may range from $100 to $600 a day, w more. Perscmal 
interviews with participants w others can cost between $100 and $500 
per interview, depending on ccmsultant fees and the ease with which tte 
intervkwee is locate aixl the informaticm collected, 

Scmietin^ reduced fees or in-kiml contributions are available, sub- 
stantially altering the ccKst framework for evaluatiim. A ccmiprehensive 
a^ssn^t should inclucte al! costs im'nc by all organizations support- 
ing tl^ evaluation, not just the direct moi^tary expetises to the main 
spcmson (lH>r a diffeientperspecU\^ on evaluation costs, sc^Alki^ and 
Sohnon 1983.) Such an approach ensures that various projected contri- 
buticms of different fuiulers and sponsors are recognized. When in-kind 
resources, such as internal staff time, computer time, administrative 
oveihaad, ard materials, are shifts to an evaluaticm project, they should 
also be fully costed (Hit In some cases, it may be more convoiient and 
meaningful lo cost cnit some cc^ts in other than dollar terms, e.g., staff 
hours to be donated to tl^ evaluation. Examples of various evaluation 
expenses are shown in chart 

Quantifiable ci^ts, such as labor and materials, are only part of tl^ 
to^ equaticm; these must be consicteied in ccmcert with other, less 
(^finable expenses. Examples of this more elusive spending categoiy 
might include the level of anticipated program disruption caused by the 
evaluation or resource losses associated with an inexperienced staff. 

Some of these nomiuantiriable outlays can best be assessed in terms of 
comparisons across different evaluation strategies being weighed. 
Consider the strategy of using in-house staff vs- outside consultants. In 
some cases, the former may be much cheaper, but the lesults less credible 
to important fuiKiers or decisionmakers. Although not measurable, the 
potential price of reduced credibility and utilization is nonetheless 
important to the overall calculation. 

Tb^ costs of various evaluation strategies are most meaningfully 
interpreted in terms of comjw^ve benefits to be derived from each 
strategy. However, evaluatira tenefits are far more resistant to com- 
parative calculation than are expenditures. First, most potential evalu- 
ation ber^fits are intangible or difficult to measure. Hie primary benefit 



3 J.} 




338 



Evaluating Social Pn4>lenis 
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r^Mance to e^Qiuaton 
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r^idts 

Po^i costs 



of evaJuation is to gain better information about JTPA; wtethcr that 
infoimation is well-utilized and leads to program improvements will not 
be known for ceitain. 

Iji addition, the evaluation process may lead to certain organiz^icmal 
enhancen^ts, or indirect benefits, which are often not considered 
because they are not explicitly connected to JTPA goal achievement For 
example, effective evaluation planning with many users may resuh in 
improve inter- ami intra-agency ccmimunication and/or cocHdination in 
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the future. Evaluation implementation may result in enhanced MIS or 
other data collection improvements. Although these kinds of benefits are 
generally not quantifiable, a thorough checklist of potential direct and 
indir^rt advantages provides a richer context for decisionmakers to use 
in weighing the cost of evaluation alternatives. 

Acquiring Resources 

While JTPA legislation supports various evaluation activities, no 
specific funds are allocated to this purpose. As long as administrative 
ftirais lem^n so limited, finding financial sup]:wrt for JTPA evaluation 
will be a fundamental concern for most states and SDAs. Decision- 
makers and planners must think broadly and creatively about funding 
possibilities. 

Several general planning assumptions underlie the various funding 
pimibilities discussed below. First, it is assumed that JTPA 's orientation 
toward interagency cooidinalion and puhlic-privaie collaboraticm sets 
stage for exploring new funding jwtnershij» for evaluation as well 
as for other program ^tivilies. A corollary to this assumption suggests 
that oilers outside tlw; JTTA system are interest^ in cvalualirai specifi- 
cally linked to program improvement and may be open to requests for 
assistance. 

Second, wh^ver fimding strategy is ultimately considerw!. those 
who will use the evaluation, if they are truly committed to the process, 
represent ihe logical source to approach for ftmds. While key users 
themselves may not have direct access to fiiiHling, they may help in other 
ways to contact sources, provide staff for the fund search, or deliver other 
valuable in-kind services. 

No funding strategy should overlook potential sources of in-kind 
contributicms for the evaluation effort; a far brMder range of supporters 
will be able to give noncash contributions to the effort. The strategy may 
inclucte a{^^^hing multiple COTtrilmtors for different kinds of support. 
Furthennore, funders will often be more interested in supporting a 
project if they see that others are contributing as well. 

Finally, the fund-search process is a capacity-building undertaking for 
the oi^anization. It requires creative program marketing to pcMential 
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supported and provides opportunities for strengthening networks of 
important contacts in and out of the program. 

Internal FitiMng and AssisUmce 

State JTPA agencies will probably lave a more centralized and 
developed research cap^biUty than tteir local coumerf^rts. As states and 
SDAs better define their respective roles and intcrreUrtiraiships, imrreas- 
ing opportunity for cooperative state-local evaluation »:tivities will 
arise. Many states are in a good position to offer valuable technical 
assistance and special services to SDAs contemplating JTPA program 
evaluation. 

In addition to state-local cooperative efforts, both aihninistrativ^ 
levels might want to explore funding leveraged from other ageiKies or 
governmental units participating in JTPA (e.g., local welfare offices, 
economic development agencies, city community develt^mem offices, 
etc.). Evaluation supported across agencies can focus more on issues of 
administration and service coordination of shared importance to con- 
stituent funders. With joint fui^ing for a p^icular or ongoing evalu- 
ation, an independent evaluaticm unit may be acceptable to all {»rties. 
Whatever the arrangement, evaluatitm activities will have to answer 
varying needs and provide recognized bci^fits to all {»rticipant$. 

External Funding and AssisUmce 

Evaluation funding possibilities exist beyond government funding 
sources connected directly or indirectly to JTPA. State and local 
agencies, however, have historically been reluctant to tap outside re- 
sources. Finding and approaching the» other funders requires staffing 
and time investment for busy administrators, which initially disa>urag^ 
organizational risk-taking. 

Ultimately, however, casting a broader net into funding realms beyond 
the familiar pays off in many ways. Even if adventureous searchers are 
not rewarded directly with the cash support they seek, their efforts will 
prove valuable in a number of ways: nonmoMtary contributions, 
increased contact and interaction within the Inis^ness, ^^^mi", and 
profes.sional communities, increa^ program visibility and crwiibility. 
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ami expam^ possiMHties for future fumling. Hie remaimter of this 
section outline some of these alternative fiinding pc^ibiiities. 

Academic institutiom often offer unique evaluaticm resource at 
ledwred costs. Faulty are a major resmirce; they frequratly have tt^ 
specialized research expertise i^ected, and are often available cU a lower 
salary than private consultants with similar expertise. Through their 
institutional ties, f^lty are better able to leverage related rescnirces, 
such as research materials* computer expertise, or otli^ faculty and 
Oudents. If tiie consuhant time required of a faculty member is below a 
certain amount, ao^mic institutions will often reduce or wmve the 
indirect costs they normally charge. 

State-su{^port^ educatiimal institutions, including community c^l- 
leges^ are part of tte state agem:y network. Their public-sector status 
provides m qpportimity ami rationale for developing c}<^r ties that can 
be mutually beneficial to both parties. In terms of hiring a JTPA 
evaluation consultant, contracting with state-supported colleges or uni- 
versities may be simpler, less formal, and involve lower indirect costs 
than would other contr^ting arran^ments. 

Students are another potential source of support for evaluation. 
Frequently, graduate stmtents are willing to devote research time to an 
outsiite evaluatiOT prq^ in order to gain pnunical work experience or 
develop materia] for a thesis doctoml pro^t. Many profession^ 
graduate sciKX>ls em:mirage or require their students to engage in such 
research. Sometimes students, as well as faculty, can partially or fidly 
support their evaluation research activities through research assistant- 
ships^ postdoctoral fellowships, or individual research grants. Although 
limited, feckral *vork-study funds do exist at the graduate level. These 
funds pay a porticm of the wages going to a work-study .student. An adcted 
benefit is that stuiknts bring the advice, interest, and suppoit of supervis- 
ing faculty, who can act as an additional quality cc^itrol on stuctent^s 
work, and who themselves may be willing to play an active role in the 
eval'iation effort, contributing specializoi expertise. 

In some cases, graduate departments or prcfessional ^hools may 
partially or fully fund studies of evaluation issues of special relevance to 
their faculty and students. One local JTPA evaluation, for instance, was 
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laigely sponsored by a nearby uni versify 's graduate business school. 
Faculty and SDA staff planned th^ wV^uation; students collected and 
analyzed data under faculty supervision. 

A number of nonprofit Inisii^s, labor, professional, social service, 
ami p»iblic interest organizaticms are interested in imiwoving employ- 
ment and training programs. A JTPA evaluation can capitalize on this 
interest in a number of ways. Members of such grou{» can serve as 
f(»Tnal or informal ^visors to the evaluation planning process and offer 
reduced fee services or provide certain resources in exchange for public 
recc^ition of their contrilnitions. 

Private foundation support used to be almost entirely the preserve of 
educational institutions and nonprofit organizations. Increasingly, 
however, public agencies have broalened their funding strategies to 
include soliciting foundations for support. Finmdation backing, as with 
other sui^xnt, is not limited to dirrct services. Many toumlations are 
C(Micemed with developing innovative ^pro^ici^ to service delivery 
and are willing to fund applied research activities in a number of service 
arnas, including employment and training. 

Most major metropolitan libraries carry standard directories profiling 
the laiger national and regional foundations and their contrilnition 
patterns. Regional directories of state and local funders me also usually 
available. These references provide initial information needed to iden- 
tify funders who are most likely to be interested in enhancing social 
programs. 

TTw major directories include fairly (ktailed {Miofiles on fouiulation 
activities (previous funding patterns, kinds of costs covered, special 
requirements, current rrcipients of support), which help the researcher 
quickly narrow the search effort. Financial reports of foundations, 
charities, and trusts within a state also provide useful informaticm on the 
kinds of individuals and oiganizations they fimd and tl^ir funding 
philosophies and agendas. Ttese repcMts are generally available through 
a state attorney general's office or the state agency that oversees the 
financial reporting of charitable organizations. 

Such funders may be more attracted to programs that are irmovalive or 
can serve as demonstration models for other programs. Evaluation of 
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programs geared to special populations (e.g., youths, ex-offenders, 
welfare recipients, older woriceis) may also interest ceitain fumlers who 
c^rwisc would not want to become involve with JTPA evaluation 
activities. 

Foundation size and location are often important considerations. 
Smaller and more local fcnindations may be unpredicuble in their 
outlook, but they will be more ^ared to local acton, and interests. They 
may support an especially appealing local project outside of their usual 
framework. 

In ccmtrast, lai^er, national foundaticms are more bureaucratic, enga^ 
in a very formalized selection of issues to be funded, have more 
specifically defined application procedures and fixed funding parame- 
ters, and apply more rigid criteria in making funding decisions. Latter 
foundations tend to have lengthy time frames for review and final 
deci.sionmaking. The L'sieofF is that major fouiKiation support, while 
more competitively sought and more difficult and time-consuming to 
thieve, offers larger pots of money, greater prestige, and increased 
likelihood of supplemental funding in the fumrc. Therefore, while an 
SDA's best chances for funding may be at the local level, the fund seeker 
should not automatically prelude national and state soun^. 

JTPA envisions a close woiicing relation between government and the 
private sector to better connect those who are being trained with those 
who can offer jobs. In tlw interest of learning more about ami improving 
current JTPA operations, the public-private partnership might ai:guably 
be extended to im;lude joint support for evaluation activities. 

Large companies utilizing JTPA services such as QJT may be particu- 
larly receptive to requests for assistance in evaluating and improving 
those services. More support may be available if the company also views 
its participation in terms of public relations returns. Although local 
scrvire agencies may be unaccustomed to approaching tte private sector 
directly for help, a mechanism for making such contacts is built into 
JTPA through the PICs and state councils. Council members often have 
extensive busii^rss and community tics, and are in a good position to help 
pluming staff identify not only who should be approachol, but how they 
should be apprc»ched as well. 
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In ^ition to a{^[m)aching businc&s contacts through JTPA chamois, 
cMl^ sources of information on private sector comf^es are available to 
help in tte fimd %arch. State employmoit ageiK^ies* eomomic ctevelop- 
ment or;i^zaHons« ami private res^rch ccnnpanies oHoi publish infor- 
mation on tfic laziest «nploycrs in the state. Also, major university and 
public liharies in each state usually carry referemre guides on corpora- 
tions and tl^ir endowment programs. 

Local companies can be cont^ted directly for information about their 
fimding interests and requircn^nts. Funding propels usually are not 
r^uiied to be as Icmg w complex as those of crther fuTKler^ and (tecision 
time is much idiorter With major naticmal i»r{K>nitions, tl^ scenario can 
be quite diflferent. Tl^y may have sp^ial (usually iHrnlocal) corporate 
ccmtrilnition units that handle ail furding reqi^ts^ often requiring more 
sojdusticated and detail^ proposals. Although these special imits may 
make the final selections, local coqxirate branches may also wish to be 
involved in the review process, and may have influence over the ultimate 
corporate funding (Vision. 

Examining Staffing Opti<ms 

Concommitant with efforts to olHain adequate evaluaticm resources, 
tl^ planner must ikvelop strategies for the optimum use of O^se means. 
A key source will be the evaluation staff. Because each organization will 
have its own evaluaticm interests aiKl i^^s* every evaluation effort will 
be unique; no single staffmg pattern suffices for all In some settings* an 
in-house team of specialists is most feasible; in other contexts, an outside 
consuhant may make more sense. An important consickraticMi is wl^ther 
available tn-house staff has tl% technical skills to aa:romplish the re- 
quired evaluation tasks. In addressing this consideration, the planner 
must fim consider specialize faffing r^uirements tl^ evaluation 
might entail. 

Comprel^sive evaluations will probably require evaluation special- 
ists in areas siurh as research ctesign and statistical analysis; mc^e scal^- 
down tffijns might manage with fewer expert sources, acting in a more 
limited consuhant fashion. Whatever the scale, most evaluations will 
require some special staffing, as suggested by chMis 5.6 and 5 J. 
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At first glance, the !»pecia!ized staHlng needs listed in chaits S.6 and 
5.7 may seem formidable. The list is offered, however, to suggest the 
kiiuis of special advice that may be needed to sustain the technical 
competency and ultimate utility of an evaluation. The experts listed in 
the second chart are necessary only if the evaluation questions present 
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special research challenges to which the core staff cannot adequately 
respond. Alternatively, a small core research staff can be constructed to 
inclwte peq>!e with such specialized skills, reducing the cost of contract- 
ing out for such expertise. 

There are two major staffing ctHifigurations for carrying out evalu- 
ation: in-houFe staffing and outside consultant staffing. Each has its 
benefits and drawlMicks, which will be more or less {mmoimced depend- 
ing on the particular evaluation context. 

In-House Staffing and the Use of Consultants 

ScMne states and SDAs can meet the JTPA evaluation challenge 
through creative in-house approaches. Although many SDAs or their 
CETA prwtecessors Iwve not conducted comprehensive evaluations of 
their employment and training programs, they often have »xess to 
uniai^)ed resources sufficient for such an undertaking. In laiger oigani- 
zations, although requisite staff are scancred throughout JTPA or local 
government systems, these resources can be drawn together as a special 
interagency cvaluatiim team, or loosely coordinated as an in-house 
consultant j^nel. 

Certainly, cost is one of the most compelling aigumenls for seeking in- 
house expertise. However, in certain i^ttings such an approach may 
involve many hidden outlays that must be entered in the overall caltula- 
tion in deciding which staffmg strategy to pursue. To locate and involve 
special evaluation staff may require significant organizational effort. 
Division or agency heads are likely to be skeptical and resistant to 
loaning personnel, un<terscoring what has been said earlier about the 
importance of building te-oad organizational support for evaluation. 
Moreover, pooling in-house staff resources requires extra management 
J taff to bridge the communication and coordination gais thatarise.Finally, 
inefficiencies associated with less experienced and less specialized 
evaluation staff attempting to negotiate a learning curve are time- 
consuming and expensive. 

Cutting comers on evaluation specialists may cost the organization 
more than the fre that would have originally been spent on consultants. 
Where in-hou.se evaluation staff lacks requisite technical expenise, the 
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great risk is that tte information otxaincd will l^k sufficient reliability 
or validity, ami fimiings will be of diminish^ value. Am>ther danger 
in using in-house evalualors may be lower credibility for evaluation 
results. 

On the other huid, the in-house ^roach to ev^uation carries with it 
son^ potentially important benefits: 

1 * Staff familiarity with the organisational setting, data collection sys- 
tems, staff cai^bilities, time sctedules, program procedures, etc. 

2. Fe^wr entry problems for evaluation staff, more rapport with pro- 
gram staff, greater receptivity to progr^matic r^eds of staff, 

3. Potential cost savings through closer monitoring and control of the 
work in progress. 

4. Opportimitics to foster inter- and inlra-agency communication. 

5. Capacity-building for further evaluation efforts. 

6- Flexibility in reassigning noiKontract evaluation staff to evolving 
tasks. 

in-house staffers also help maintain the momentum of a user-centered 
evaluation. If they become involved and invest^ in evaluation in the 
early planning stage, they may be more committed to facilitating or 
encouraging the active use of the nesults. Critics of the in-housc 
approach argue that even if these resources are available, some important 
potential benefits offered by outside consultants should not be over- 
looked Tl^se potential benefits iiKrlude: 

1. Greater credibility with evaluation users, particularly funders. 

2. Separation from the oi^Miization, which allows for greater objectiv- 
ity and fairness (actual or perceived), 

3. More acceptance from program staff who feel less threatened 

4. Greater assurances of a quality product produced by an experienced 
specialist. 

5. Ability to allow staffing levels to fluctuate in response to varymg 
resource needs. 

Outside evaluations may be most appropriate where organizational 
tensicHis or mistrust call for an evaluation with maximum separation 
from thte JTPA system. For example, outside consultants may provide 
greater credibility when the evaluation calls for a more subjective 



348 Evaluating Social Pn^temt^ 

as&^ssn^nt of proce^^ or impleir^mation factoid. In such a case, servio^ 
pa /iden^ SDA staff, and otlwr stake-hokters may more easily trust and 
accept tte inter^n^tive evaluation results of an outsicter. 

A comprcmiise stafltng strategy involves the judicious use of consult- 
ants at critical planning ami implementation juiwtures of tl^ evalumion, 
whm expertise is most needed. For example, a consultant might be 
brcnight in solely to a^^ess the evalimhility of a program or to ctevelop the 
exalmition design^ which others may carry out. A cmisultant can 
CQimbute by perfc^ing th€»;e tasks mi^t associated with ob^tive 
judgment: the research design* tte data analysis, and iSx evaluation 
repxt« Alternatively, a consultant's role might be strictly advisory, 
limited to reviewing am! commenting on the in-housc evaluation work 
in progress. In this maimer, quality contix>l can be assured, while 
consultants' fees are contained. Wl^n a fwmal review is comJucied by 
a completely imtependent {wty, the process is consider^ an evaluation 
audit. 

An evaluation audit by an independent third party serves several 
functicms. An ai^itor can formally review and critique the evaluation 
plan as well as the implementaticm procedures and tt^ final evaluaticm 
report. By reviewing the plan before evaluation commences, the aiKlitor 
can spot problems, gaps, and weakmsses in ihc plan and ^ggest changes 
to improve iht scientific soiUKiness, the oi^anizational effi^tiveness, or 
the efficiency of the evaluation. 

Using an outside evaluation auditor improves tl^ utility and appropri- 
ateness of the evaluation, and enhances the credibility of an effort 
plann^ and executed by in-house staff. Because the use of an audits 
offers many of the protections of contracting-out, at reduced cost, it is an 
attractive staffing alternative. 

Conclusion 

Technical concerns about planning and implementing an evaluation 
often overshadow organizational issues, tnit organizational fsKrtors can 
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treimiwkHisIy influence itie evaluation proceKs and the ultimate useful- 
ness of ll^ evaluaticm resultn. In a user-centered a}^>n}»;h to this 
{banning, organizational context is the primary focus for all evalu- 
ation planning ^tivities. Thin context ^^fines who the key users of the 
eviduation will be and how the evaluation must be gerarally molcted to 
meet tl^ir information and other program-related n^s. Users and their 
ne«»}s drive the evaluation from preliminary planning to ultimate dis- 
semination of results. 

Oiganizati(^al planning to support cvaluaticii also places importance 
cm (feflning ami engaging additional key actors, such as program staff, 
research staff, cmnputer and data techniciaas, and evaluaticm fumlers, to 
work with one another in a coordinated fa.«hiori. The collaborative 
involvement of all f»uiicipants in the planning process is important on 
many levels. Collaboration on evaluation aeates i^w communication 
pathways acn^ traditicmal organizational divisitms arKi helps overcome 
organizational isolation. It fosters cross-fertilization of ideas regarding 
what is impcmant to study in a program and 1k>w best to undertake this 
effort. Collaboration encourages greater organizational support for 
researchers so il»y can better protect and enhance the technical compe- 
tence and reliability of the evaluation results. By the same token, this 
appn^h sensitizes researchers lo user perspectives at all levels, ensur- 
ing tluit tte research approach selected truly reflects users' needs. 

evaluaticm planning process requires tl^ plarmer to play a particu- 
larly strong, proactive role in creating organizati(Mial supnort. In addi- 
tion to identifying fH>tential users ai^ other key actors, tl^ plani^r 
(kvelops strategies for maintaining their involvement and interest. He or 
1^ may have to market the evaluation actively both inside and outside 
of the immediate oi^anizaticHi and help potential users create new and 
mutually bei^ficial partnerships in support of evaluation. The plarmer 
may be an oi^anizaticmal ground-breaker in developing new communi- 
cation and coordination lirtks to facilitate evaluation implementaticm and 
dissemination. And Anally, the planner may have to assist in or direct a 
cn:aiive search for new evaluation resources. 

Proactive planning translates into a special Challenge for thc^ in- 
volved in the process. Bringing together people from different perspec- 
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tives and experiences to play new roles and perform new tasks entails a 
dtegiee of uncertainty ami risk>taking for everyone. In <^}enly imjgniz- 
ing this challenge, the evaluation planner takes tte first step in incorpo- 
rating tJw organizational context into the evaluation planning process. 
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Strategies for SupfMirting 
Comprehensive Evaluations 

Ann Bonar Blalock 
WashingttHi Employment Security Department 

The active-reactive-adaptivt evaluat(>r nvrks with decision imkers 
to design an evaiuation that incMes any and ail data that will help 
shed light im evaluation questions, given constraints of resources 
and time. Sitch an evaluator is amunitted to research designs that 
are relevant, rigorous, understandable, and able to produce useful 
results that are valid, reliable, and believable. 

Michel Q. I^tton 

Qimiitaiive Evalmtion Methods 



In the last chapter, the author discussed important aspects of initiating, 
planning, and implementing state and local evaluations from an organ- 
izational and political viewpoint, hi the chronology of user-oriented 
planning activities associated with program evaluation, informatitm 
dissemination and utilization represent the final set of related planning 
and research respomibilities. Communicating evaluation information 
that is usable by decisionmakers, and at iImw^ points in tl« decisiomnak- 
ing process where receptivity to such information is likely to be the 
greatest, requires a conscious effort on the part of evaluators and agency 
planners woiking in j^itnership with thwrn. The author made it clear that 
dissemination and utilization must be anticipated and dealt with through- 
out an evaluatiffli effort. Doing so increases the relevance of the 
conclusions -..*sed on research findings and the recommendations for 
action that flow from tl^se ccnKlusions. 

Information Dissemination and Use 

The political process tends to resist research as a source of informa- 
tion, unless it can be used to support a position alre^y formulated 
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(Undblmn 196^). Buiraucmcies arc tlM^ foundation of modeni soc 
ties, tHit one of their emiuring traits is resistani^ to change, which is a 
pc^ntial outomie of program evaluation (Pierrow 1 979). Hie ac^ctemtc 
training ami profession^ norms of evaluators predispose them to use 
technical language in explaining evaluation plans and results. H^se 
realities have affected the ability of evaluaticffsponson** planners, and 
researchers to influerK^ the policy process. 

Even though well-uncterstood by program practitioners, the ^oigan- 
izaticmal politics** of evaluation has only recently been given serious 
attention by those involve in evaluation research (Cront^h 1980; 
Qielimsky 1987a; Palumbo 1987). And mmt of what has been written 
about this phenomenmi has been ^plied to national-level evaluation. 
State and local environments differ regarding the specific political and 
organization influences operating, but commc^ issues surface at ail 
levels of government regarding the most effective context for dissemi- 
nating and using evaluation information. 

For example, a 1987 exploratory survey of state ami local program 
directors and oversight staff in JTPA yielded insights similar to those 
refkcted in the new national literature on evaluation politics 



(Blalock 1989). There was a consensus among respondents that the 
following conditions were essential to tli^ successful communication 
and use of evaluation information: 

1. Evaluation questions must be framed in such a way that their 
answers will be policy relevant. 

2. I^>licyniakers, planners, ami managers must be invested in these 
evaluation questions and the use of evaluation results fiom the 
beginning of the evaluation planning process, j.e., evaluation must 
be perceived as meeting their ctecisionmaking nmls and interests if 
they arc to commit i^ed^ evaluation re^urres. 

3. Determining how the new informati<Mi from an evaluation is to be 
used is a {H^requisite to deciding how feasible it is to condinrt the 
evaluation, i.e., determming wtKitt^r this information will be used 
for ongoing operational management, short-tenn policy decisions, 
or major long-term policy shifts. 

4. Decisionmakei^ must be involve at some level with the develop- 




Strategtes for Su{^x»ting Ccmiprriiensive Evduaticms 3S5 

ment of evaluation plans axni strategies for using the results for future 
planning ami im^gram operation^ i.e.* Uiey must be kept in the review 
and approval loop. 

5. Top numagement must be involved to some <tegree in the evi}lution 
of evaluation plans and ^ivities, 

6. To be coi^icfered credible, research designs must be as unbiased as 
pc^sible, ami the sources of data as accurate and reliably collected as 
can be m^mplisl^ within existing resources. 

1. Evaluaticm activities must be presented to program staff in a positive, 
nmipunitive, mmthreatening way to assure tli^ir acceptance axKl co- 
opermion, i.e., presented hoti^tly as a way to make pnK:tical im- 
provements in policies and programs, 

8. Program oversight staffs must have more support for and access to 
intensive training in the specialized skills r^juired for both evalu- 
ation planning and program evaluation. 

9. ^ ^on findings must be timely, directly aj^licabieio programs, 

jsented to users in ncmtechnical language. 
Aji««.K/ugh this survey reported a surprisingly high level of evaluation 
activity <Mi tte part of most states arrf a large number of local Krvice 
organizaticms, it also reveal^ a candid litany of barriers to bringing 
program evaluation mto the JTPA system. The following were tte issues 
of greatest ccmcem: 

1 . Lack of commitment to evaluation as an ofganizaticmal goal, fmucity 
of managen^t directives supporting evaluation, hesitance of 
managers to raise evaluative questions, and difficulty in gaining 
acceptance of evaluation as an integral oversight function and 
practical management tool. 

2. Difficulty in interesting state and local policy councils in evaluation 
and in generating qt^tions of potential use in carrying out it^ir 
policy development, ctKmlinatton, and oversight responsibilities. 

3. Insufficient funding, staff time, and research expertise. 

4. Unclear (tifferentiation betw^n mfmitoring and evaluation in de- 
signing and implementing evaluations, ami difficulty in discarding 
a ""monitoring mind-%t'' that associates evaluation results with 
program smicticms. 
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5. Intra-oTganizaticmal bureaucratic territoriality, competition over turf, 
fear of Ic^^iing control over programs^ and displacement of evaluation 
goals by organizational agendas. 

6. Problems with data leliabiHty , i^rcess^ and confidentiality, and a lack 
of imaginativeness in n^i^ing data from multiple data l^ses acravs 
programs* 

7. Concern about building the necessary collaborative lelaticmships 
with other agencies* and assuring that important constiti^ncies have 
a sufficient sense of ownership in evaluaticMi efforts to support an 
appropriate use of evaluation fmdings. 

In gei^ral, the most disturbing issi^s v^re evaluation funding, the 
leseaich sophistication of evaluation efforts, and tl^ir neutrality* Im- 
plied was an unck^rstandable resistance to moving beyond tr^itional 
program perimeters and tf^ir organizational contexts to seek funds, 
acquire expertise, and ^velop i^w ways to make evaluation relevant to 
the policymaking process. 

This view}M)int on the evaluation challenge reveals them^ ccmimon to 
otii^r social pn^rams comreming the organizational status of evaluation, 
evaluaticm commitment and (^ability, evaluation lesources, and the 
utilization of new ob^tive information. Such themes are not riew. 
Those involved in knowledge production have always faced fundamen- 
tal problems of this kind While r^pgnizing that such problems exist, we 
are entering a particularly exciting period in terms of the application of 
scientific tools to the study of state aiKi local programs. 

The scientific and technological l^se for program evaluaticm was 
never stronger. Ttere is a new legislative oversight mandate in many 
programs* which supports ev^uation. States and local areas have greater 
ccmtrol over tli^ resourcres thii reinforce such a commitment, and there 
is growing acceptar^e, interest, ami activity at the state and local level in 
establishing an evaluation caj^bility. State and local program evaluation 
is a rapidly evolving phenomenon of considemble significance and 
pc^ential utility. H^ref^e, it is important to give attention to seme 
dmnges that might enhance tl^ ability of states and local program 
OTganizations to meet tte evaluation challenge. 
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Strategy for Meeting tbe Chalten^ 

Unflattering myths persist in the practitioner community about re- 
seaich, but the assumptions that keep these myths alive are weakening. 
The uicieas^ emfdiasis on cost-efficiaK:y now supports the analysis of 
KHitinely collected prognun infonnalion to answer more complex and 
useful questions than are posed or can be answered by prognun monitor- 
ing i^ivities. The more visible influence of tbe private sector in program 
decisicMimaking has brought with it the rtetoric of industrial quality 
omtFol and product research, which has indirectly supported the obj^- 
tive study of social program processes and their outcomes. 

C(nigressi(»ial pressure on states to assume new over^ght resp(msi- 
bilities as a tradeoff for increased power has led states and local areas to 
consider how to ^^commodaie the evaluation fuiv:titm. The risks in 
having only subjective informatira to offer to decisicmmakers now 
frequently outweighs resistaiKe to mcninting scientific studies. The 
t^ous witli^ring away of myths, however, is not enough. Changes must 
be made in how we view the rale of evaluation, the way we acquire 
resources to sui^rt and use it, where we locate it organizationally, what 
range methods we i«ed to consicter in studying evaluation issues, and 
the kinds o/ expertise we must acquire to conduct comprehensive 
evaluations. 

Redefining Expectations about the Role of Program Evaluation 

In this book we have repeatedly emphasized that the major role of 
evaluation is to impmvc programs as strategies for resolving social 
proMems. But we must guard against overstating the ends that evalu- 
ations can accomplish in terms of affecting the ccMwlusions of key 
(tecisionmakers in tte policy pnxress. Exaggerating *he role re^arch can 
play merely sets up evaluation sponsors am) jmn i nts for a chnmic 
sense of failure regarding tte impact of research efi FurthermtMie, it 
can di<itract research «ivocates from identifying and utilizing more 
realistic oj^pKmunities for affecting program derisions. 

An overly dogmatic and inflexible view of what can be accomplisl^ 
with evaluative informatiim can also deflect policymaker's attention 
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finom the comptexity of social problems — the multiple causes that must 
be addressed, and the myriad interrelated program alternatives that must 
be marshaled to resolve them. We need to develop a less territorial, more 
pragmatic, and shrewder view of evaluation's role. 

An adjusted (tefiniticm of the role of evaluation might more wisely 
view decisions as the rrault of debate among many actors, the outcome 
of negotiation md sometimes ccmflict. As Cronl^h commits, 
"Action is determined by a pluralistic community" (1980). No one 
evaluatimi or scries of evaluations le^ily supports one right decision. 
Too many important questions for making an ideal decision have to be 
left unanswered, or are unanswerable even by scientific means. As Carol 
Weiss suggests, there are only "best comprwnises" (1988). Also, deci- 
sions frequently bypass formal ctecisionmaking processes. Hiey are 
often the by-product of a progressive, largely nonrational narrowing of 
existing alternatives — a "nonlii^ar prtx^ess," according to Gary Henry 
(1987). In this context, we must appreciate the significant and more 
subtle indirect effects as well as tte more immediate and tangible direct 
effects of evaluation Mrtivitics on the nature arel ctmtent of social 
programs. These quite different kinds of influence may be equally 
pervasive. 

Hiere is little question that certain evaluation ccmtexts and situations 
produce immediate policy or program effects. In cases where decision- 
makers have requested a specific evaluaticHi for the purpose of making 
immediate changes within a j^icular thro frame, for instance, or where 
the underlying agenda for conducting an evaluation is a clear demand for 
and commitment to change, there is often a ' easily describable 
influence for evaluation. There are situati« which the policy 
question is of fundamental interest to the inten^' . 2r, and the evalu- 
ation findings clearly answer that question. 

Findings can lead directly to legislative option, i.e., program reauthori- 
zation and new program rules and regulations. Synthesizing findings 
from past evaluati(Hi$ and applying this analysis to a high-priority subj^^t 
area can sway decisions. Over a particular year, tJ« U.S. General 
Accounting Office produced 290 evaluatitm reports with 1,135 recom- 
mendations. A study of what happened to these recommendations 
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reveal^ that 80 percent were acceptol by the feckral agencies to which 
they were directed (Chelimsky 1987b). 

Other contexts and situalion.v-conceivab!y the majority—involve 
iiulirect infli^nces. Evaluation information may ^pe asstm^tions 
subsequently taken for granted by political and organizational decision- 
nnakers in the imx^ of negotiating decisions, such as assumpticms 
about the nature of tt^ problems to be resolved the characteristics of 
thme to be changed, tl^ chan^ strategies considered tcmx effective. 
This information may influence the design cf ney\' programs through tte 
^xumulation of evidence from past research. It may infli^nce the 
language of the policy debate — reporting empirical evidaicc to supptm 
positions lemis credibility and power to those positicHis. It may have an 
effect as a rationale for change, circumventing more obvious barriers to 
the utilizatitm of evaluation information. It may ejqmml the policy debate 
by including a broader range of alternatives to ccmsider. It may set 
parameters around the debate, such as which issues are to be given 
attention, and how they are to be defined and prioritized. It may reorient 
policy agendas by suggesting which program implementation theories 
may be flawed and which reasonably effective in producing the desired 
results (Cmnbach 1980: Chelimsky 1987a). 

Evaluations can also alert decisionmaken to an immediate social 
crisis or to troublesome long-term trends requiring a policy response. 
They can help provide a new framework within which issues are 
c<msidered, supporting innovati(Hi in policy development and program 
design. They can contribute information that supports tl« formation of 
coalitions that do wield direct persuasive power over the policy process. 
In rethinking tb; role of evaluation in the policy process, we may have 
to relinquish the niive expectation that evaluation must always have a 
promii^nt, direct, measurable, and immediate imj^ct on policy and 
program (tecisions. 

Devek^ing New ApprtM^hes f<N- ^curing 
Evaluation Support, Resources, and Utilization 

The impressive evaluation efforts now occurring at state and local 
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levels have, neveitlwless, been cimstraii^ by depcinknce on scan^ 
resources within the pit^rams being evaluated. The JTPA survey, for 
instaxu^, imiicat«i that mmly all state and local oiiganizatiras paid for 
evaluation activities exclusively with JTPA alministrative funds, a 
small pot of money. Only two states Ymd ever leveraged funds arross the 
programs to be coordinated with JTPA. Only one had acquired fumiing 
outside the JTPA system. Most agencies used in-house JTPA staff in 
program monitoring or management infonnati(Mi system (MIS) units to 
design and implement evaluations. Remarkably few engaged in joim 
cross-program evaluations, reli^J on research consultants, or used a 
combinalifm of in-house and outsi(te research expertise. 

Chw wtnild suppose that ccmcem about a lack of ^i^ral support for 
evaluation within JTPA and the recognized need for new in-depth 
information would have led to more imaginative efforts to combii» 
resources from multiple sources. Yet, it was clear that in most cases 
states and local areas had not fully exploited chances to expand their 
resource bases. Although survey respomtents were concerm^ about 
evaluation utilization, it was obvious that minimal eneigy had been 
invested in systematically anticij^ting utilization issues throughout an 
evaluation, or in advoc^y arKl marketing activities mce an evaiuaticm 
was completed. In a period of reduced resources in the 1980s, this 
revealed a confining bureaucratic jiiind-set about resource c^^xMlunities. 

This sug^sts tfie need for a new ^nse of empo vermcm at the state and 
local level that resists the tendency to rely on limited, truiitiona] smuces 
of sup^rt. There is risk in leaving the apparent safety of familiar 
organizational territoiy^ — a sen^ of loss of control in reaching out to 
other people and other programs. Collaboration involves an inevitable 
renegotiation of authority and ownei^ip. Nevertheless, new attitudes 
are liberating in terms of ex{»nding the scope of evaluation efftwts and 
increasing evaluatiiMi resources. 

Empowerment, however, will require the purposeful creation of new 
collaborative relationships — from ihe beginning to the emi of tlw evalu- 
ation process, ami beyond. States and local organizations will need to 
ccmstruct support networks that bring together representatives of con- 
stiti^ncies that arc significant sources of support in considering evalu- 
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atirai issiKS, provide a base for evaluaticMi activities, and ensure &k 
ai^m^ale use of new information. 

Rather than focusing on a lack of access to resources needed for 
evaluation, it will be important to concentrate on identifying potential 
r^ourrcs, bringing tl:^ into the social seivice system, invei^ing con- 
tributes as i^utners in efforts to make that system more accountable, ami 
offering something of value to these new partners in return. New partnci s 
can be offered public recognition, increased organizational credibility, 
mi inf(Hin^(m required to pursue tteir own goals. New linkages can 
be established, giving them greater uCotss to the resound they need. In 
building support iwtworks. ways 'o offer opportunities for a mutually 
inofitable exchange with these conji Htuencies will have to be developed. 

Experimentation with leveraging funds and other kinds of assistamre 
diould also be encouraged across different pools of funds within pro- 
grams, across different service providers who are expect^ to functi(»i 
cooperatively, and widiin a larger system of integrated services. 
Developing /n/K^Z/ig consortia to jointly fund evaluations, and voilalxf 
rative bodies to receive and allocate jointly ccmtributal resources is 
important. 

Crafting general strategies for approaching funding sources at Uw 
nati(^ level and enveloping specific strategies tailored to carefully 
selected private corporate and foundati<Hi sources should be part of this 
innovative resource ^uisition effort. To be successful, these strategies 
must be responsive to the kinds of social exchanges required to interest 
and invest potential contributors. (See Feldman, chapter 5 in this 
volume.) 

TTiese suggestions should not imply the creation of a large, amorphous 
iKtwork of people: acquiring resource fwrtners should be a hi^ly 
sel«:tive fHocess. TTie core of a support iwtwork needs to be organized 
as a formal advisory body, which consolidates tlw iKtworic*s influences 
and Iwlps maintain it over time. The siz» and purpose of such a group can 
be limited, so that its mission is not easily displ^;^. Composition can 
be confined to representatives of constituencies that wield the most 
power in obtaining evaluation resources and securing tl» use of the 
information produced. Assigning a specialize plani^ to act as a liaison 
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betAveen this advisory body and ongoing cvaluaticm activities can de- 
velop imponmit bonds among program practitioners, evaluatoni, ccm- 
tribute^;, and users, and can help them to cooidinatc their agendas. 

Building efTective oi^-^izaticmal Imkages secure mutual invesmient 
over time, so that future evaluations will have an immediate base of 
support. Formalldng a support t^twork irnrreases opportunities to gen- 
era otherwise inaccessible funds and nonmoTOiary contributions out- 
sicte the program. It prot^rts evaluation activities from being ccM)pted 
by q[>ecial interests, including thc^ invest^ in the program, and pro- 
vide a buffer when evaluation resuhs are controversial. More impor- 
tant, it can influence a pit^ram^s status in its environment by increasing 
public perception that a program is genuii^ly accounuibk:, and by giving 
evaluation special legitim^y as an accepted pan of d^isionmaking. 

Increasing the Autraomy of Evaluation Activities 
and Their Influence on Ptrfides and Programs 

The JTPA survey refcm^ to earlier ickntified a problem faced in most 
social programs — the organizational location and autonomy of program 
evaluation activities. Most JTPA evaluation activities had reportedly 
been carried out by staff within programs being evaluated, in particular 
by monitoring and reporting staff. Only a few of the larger state agencies 
and urban program orgmizattons had used evaluation units separate 
from program divisions to carry out JTPA evaluations. 

Hiis is understandable; funding for oversight activities has n^ition- 
ally emphasized mcmitoring progrmns for compliance with mles and 
regulations. The new interest in meeting formal performance slant ^' 
has increased the priority given to monitoring the allocation of scarce 
program dollars* Expectations for monitoring staff have encouraged a 
strong investment in the program being reviewed, and the location of 
monitoring activities within program divisions has legitimizftl this 
interest. This lack of neutrality, however^ can seriously reduce tte 
objectivity of evaluation activities. 

One practical rc^^lution of this problem is to relocate evaluation 
activities where ttey can be given greater organizational priority, fund- 
ing, and autonomy. One of the most useful ways to accomplish this is to 
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develop evalmikm units thai are semi-autraiomous from prc^ram divi- 
sions and serve an evaluation function for multiple programs within a 
state agnicy, cm- are given responsiMlity for a singular and separate 
oversight function within a local pxjgram organization. The directors of 
such units would be required to report lo the decisi<nimaking level of tte 
OTganization. This ccnK:ept of state or local level "general accounting 
offices" has been applied suwressfully by several states and lai^e 
program organizatioi^. 

Utis kind of unit could be the focus of network building for evaluaticm 
activities, and assun^ respwisibility for woHiing with an evaluaticm 
fuivisoiy bcxly. It could become the focal point for assessing resource 
needs for evaluation aiKi contr^ting with state universities, local com- 
mimity colleges, aral privMe research firms for acklitional expertise. Its 
t»sic funding within state a^ncies could be collaborative, drawing from 
multiple s(}urces of administrative aiKi technical assistance monies as 
well as outside sources. At th^ state and Icxral level, separate evaluation 
units could more easily attract contributions specifically targeted to 
program evaluation activities, and legitimize the acquisition of staff 
sufficiently trained to participate in designing and implementing com- 
prelKinsive evaluations. 

l^Kre is always the danger thai autonmnous evaluation units will 
beccnne isolated from program*:, exercise too much professional discre- 
tion, develop their own langwuge and agenda, and become threatening to 
proj^Tun management. However, such units could be maiKlated to 
maint^n close coruiections with progr^ divisions through liaiscm 
personnel with evaluation planning expertise, who are outstatioi^ with 
evaluation units, or through staff working within a program division and 
in close {^rtnership with the evaluation unit's rc.search ^aff. 

A significant advantage of such units is their potential for concentrat- 
ing evaluation fumi search, netwcnting, research ctMitracting, research, 
and evaluation marketing activities in one place. Their position on the 
organizational chart permits them to be direct recipients of high-level 
cfecisicms and to exert an influence on the organizaticHi's overall over- 
sight commitment and direction. Periiaps most impcntant, this kind of 
organizational niche for program evaluation assures maximum objectiv- 
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ity, encourages research ctMnpeteiKe, adds stature to the role of evalu- 
ation in the oi^anizatitMi. ami offers the best opportunity to actively 
advocate the use of evaluation information. 

Expanding the Evaluation Repertoire 

As discussal in chapter 1 . ttere has been a terelency to oversimplify 
programs in order to study them more quantitatively, and to focus on 
highly selected aspects of programs without considering how they fit 
together. There has been little systematic interest in testing the accuracy 
of the assumptions uiKierlying prc^ram (tesign. or in viewing the com- 
ponents of programs as part of a larger organizational system. 

As perverse as it may sound, increasing interest in the soci^ sciences 
over the last three (k^cades is in part responsible for these limited 
perspectives. Tliis interest led to specidization within social ^ience 
research and the professionalization of evalmtim research. It has had 
an unintended divisive effect an the research community, regarding 
which appro^hes and methods are "best"* — experimental vs. nomsxperi- 
mentai. quantitative vs. qualitative, outcome-oriented vs. implementa- 
tion-oriented. The strong traditional emphasis on net impact studies and. 
more recently, on experimental field studies has assigi^ a higher status 
to research in the policymaking community, but a price has been paid for 
greater rigor in the narrow sets of issues that can be ^Idressed. 

Even as the significance of implementation stiuJies has gair^ de- 
served recognition, most process evaluaticms are teing carried out 
independently of outcomes snjdies. This limits tlwir usefuhiess in 
offering explanations for tte results of outcome evaluaticms. This 
"eillwr-or" attribute of evaluaticm research has restricted the evaluation 
repertoire uni^cessarily. It is a particularly restrictive perspective in 
evaluating state and local programs. 

As Chelimsky suggests, when the «>ntext of an evaluation involves a 
heated policy debate, a rigwous generalizable net impact study may offer 
fHotection to the cvaluator, but be neither feasible nor appropriate in 
answering tiie question of greatest interest (1987a). Even when such a 
research design is ttK appropriate choice, timeliness may take prece- 
dence over the power of scientific rigor. Under different circumstances. 
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a ret impaa study would be imperative in answering the policy question. 
Hie increasoi flexibility ^ght in wider dioices should not be viewed 
as comimnnising scientific principles and methods. It simply allows us 
to jfully utilize ttem. 

At state and local levels, the repertoire should offer the full range of 
tlttoretical arKl nwthodoiogical choices and encourage tl^ use of ctHnbi- 
rotions of choices in uiMtertaking comprehensive program evaluations. 
We may want to combine a rigorous net im|»ct study using ecOTomeiric 
mediods with an exploratory study of jwticular aspects of implementa- 
ticm, or combine a survey to determine the attitudes of program personnel 
with a study of tte gross outcomes of pn^ram partici[^Ls. We may 
want to ccnnbine a survey of employers with a survey of clients who have 
been trained or employed by them. Caning up the evaluation process 
to moie diverse opportunities that cut across different apint)aches and 
methods can free us to study issues of more dirw:t interest to those 
making decisions about program change. 

In expanding the evaluation repertoire we can benefit from a ccmsid- 
erationof a broader set of research choices. We should also be concerned 
about the manner in which information yielded by these choices is 
C(»nmunicated to ibcts. We must be more resjMnsive to decisionmakers* 
requests to translate evaluaticm findings into meaningful form from their 
points of view. The scientific interpretaticm of results must be converted 
into a political and organizational interpretation. 

Cleaily. the evaluaior is oMigated to make the appropriate distinction 
for the user among findings, cmclusions, asvi recommendations. The 
evaiuator's primary respcmsibilily is to report findings as honestly as 
possible, with all of the necessary qualifications. There is, however, an 
important sec<Midary obligation for tlw^e working in applied research. 
They must draw pragmatic ctmclusions from those findings, if jwssible. 
and suggest effective acxwm that can be taken to improve policies and 
programs. 

In scMnc cases the findings will not warrant this leap. Even in this 
instance, the evaluaior can recommend issues to study in future evalu- 
ations. In most cases, however, something will have been learned from 
an evaluation that supports making these progressive leaps from the 
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research iiself. These conclusions and recommendaticMis will not be 
heard by the (kci&icmmaker imleiu; the evaluaior— with tKivice fiwn 
(M^tttioners — presents these iiteas attractively and in nontechnical user- 
oriented language, trims what is Irrelevant, condenses, and decides 
which fliKlings are the more important ones and which can actually be 
addressed by those making decisions (Chelimsky 1987a; 
Cronbach 1980). If evaluation information from a more varied $et of 
research opportunities is to be effective, the presentation and marketing 
of this information must also become an accepted pad of the evaluation 
repertoire. 

Using a New Approach in Staffing Evaluations 

If a broader research repertoire is used, staffing evaluations differently 
at state and local levels is essential. Althmigh some state agencies and 
local program oi^izations have cvaluaticm units whose staff is well- 
trained in research, many more depend on staff with considerable 
man£^rial, planning, or computer science expertise, but minimal re- 
search training. It is frequently assumed that these staff members can be 
formally or experientlally retrained to ccmduct evaluations, and that 
assigning staff members dual oversight roles is an efficient way to meet 
accountability responsibilities. 

It is extremely difficult for those mcmitoring program compliance to 
maintain ob>a:tivity about the program being evaluated, and it is unre- 
alistic to expect that even an intensive ^ries of courses can sul^tute for 
graduate research training. A less-than-thorough grounding in research 
principles and methods constrains staff in judging Uie feasibility of 
evaluation questicms, identifying the resources needed for sc jnd cvalu- 
atic»is, aiHi developing viable research '^igns. 

A separatirai of talent is, thCTcforc. noi only r^cessaiy Init efficient 
Researcl^rs are best equipped to carry out program evaluations. Moni- 
toring and MIS staffs have key roles to play in reviewing evaluation 
i^ues ami plans, and in opening access to program infcmr <n the 
{HtK^ss of implementing evaluations. Basic evaluaticm tr- 
extremely useful in increasing their sophisticaticm as they 
the evaluation {ntxress in that role. 
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There are several changes thai can be explored, including hiring an 
inteidiscipliiuiy r^earch team as the core of a central evaluadm unit, 
having this unit selectively use outside research consultants, and lede&i- 
ing the evaluator's role. The benefits gained in undertaking these 
activity include the following: 

1. A C(He in-h(nise research staff with expiMsure to the knowledge t»!» 
of nunc than cm social science discipline can draw from alternative 
theori^ atxnit social proNems and how they can be resolv^l, and 
frran a range of mcthodol(^es for studying social programs. 

2. Job (kscriiKicms for such a staff benefit fiiom consultation with ex- 
perienced researchers willing to conuibute tteir reviews and com- 
roents. Hie cboi^ of consultants ^htnild reflect the ratine and extent 
of education, specialized training, and experience needed to con- 
struct an interdiscipUnaiy team. Pysi^MU^l selection ^KHild focus (Ml 
a knowledge of social theory, research design, advam^d research 
methods, social statistics, and computer analysis. The applicant's 
level of expoience and success in working collaboratively with 
program prm;titioi%rs throughout the evaluati(m fsocess, including 
souring die u%oftheinformati(m{m)duc^, is also important. The 
political, organizatioml, and interpersonal skills ncaied to work 
coc^jeratively in pragmatic program settings requires the research 
team to be familiar with onKiepts in ix>litical ami oi^ganizatifHial 
ti)e(»y and behavior, and interpmonal iKsgotiation. 

3. As effective as an evaluation unit may become, its integrity and ob- 
jectivity will need to be maintain^ by supplementing this in-kind 
research expertise with assistance from outside researchers. An 
effective strategy is to contract selectively for time-limited, special- 
i^ expertise require by a particular evaluaticm, i.e., expertise 
needed but not available in-house, w expertise that needs to be 
oHair^ from a soun% cleariy seen as politicly neutral. 

4. Tl^ cost of hiring researchers ami ccmtracting fw specialize re- 
search expertise are important considerations. This expertise, how- 
ever, is not as expensive as most states and local oi^anizations may 
suppose. Many af^lied researchers are accessible to states and local 
areas through mte community «>Ilege and university systems, other 
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state higher education instituti<ms and related re»»xx:h institutes, and 
private rewanrh organizations, Ttese evaluators are often eager to 
utilize oi^>ortunities to v^oik in applied %ttings aiui to make ^\Kh 
experi^u^ available to grEduate stuctots. A social exchange, as 
well as an excl^ge of Ascal benefits, is involved. Also, in servicuig 
more than one program, an evali^on unit can be partially supported 
by poolii^ adminisn^ive and t^hnicai assistaiKe funds* An evalu- 
ation support network can be a critical asset, and an advisory grwip 
an important source of omtributed expertise. 
5. Thinking more creatively about ccmsmM^ting a competmt interdisci- 
plinary evaluatifm team ainl maintaining usefid linkages with pro- 
gram divisirais shcMild nc^ distract us from expaiKting tte role of the 
program evaluator. A perennial complaint is that this role is too 
intellectual and rmoved from oxganizaticHUil realities. Evaluators 
must be willing to learn to play roles beyond their primary technical 
responsibilities in tl^ evaluation process, inclmiing tti^ following: 

(a) StiKients of or^ganizations, tte policymaking process and how 
tl^ political system worlcs. 

(b) Research advocates with infomiation users. 

(c) Catalysts for listeniiig to users' concerns arKi helping them raise 
useful, researchable questions about j^ograms. 

(d) Organizatior^ team players within an evaluaticm unit, aiKi 
members of a wc»1^tng partx^rship wi^h evaluaticm planxK^ 
operating within program divisicm 

(e) Politick ar^i organizaticmal int^rim^ters of evaluation fi^^ 
ccMKlusions, and recommendations to users and the n^dia. 

(f) Evaluators of the impact of evaluations, i.e., tl!^ effectivei^s of 
evaluatiiKi efforts in influoicing tl^ direction {m>grams take. 

(g) Consummate agents of change. 

Unqu^ti<»iably, tl^ sug^^ changes will require more effective 
educaticmal efforts to explain the befits of applied scioKe, increased 
fimding fnr evaluaticm, bureaucratic commitment to evaluaMve activities, 
inventive collaboration among thc^ responsible for {Miograms, more 
:^^>histicated research ami planning e^rtise^ and greater appreciation 
of the value of acxunulating a usable bcxly of knowledge about social 
programs. 
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CoEHrludiiig TlHHights 

The auflKyrs of \hc preceding chapters luive offered di^mrtly ctiffermt, 
but complen^tary, penq^ectives chi im^ram evaluation at state and 
local levels. In tt^ three research-oriented ch^>tm« each authOT tms 
drawn fiom his or l^r own area of social science re^arch to design a 
pr»:tiad^ scientific a{^m>ach to ^iKlying a number of impratant aspects 
of programs: array of prc^ram outcc^nes experienced by ciimits and 
cHhers to be af^ed by a program, a im^ram's rxi effects, and the or- 
ganizatiiHial poHci^ ami i»:^ces that shape a prpgram^s infli^ce in 
creating tiK intencM chmi^. Nevertheless, a ctmrniitment to ccrniim- 
tensive program evaluaticMi loimins the cmtral then». There is ccmtinu- 
ing emi^msis cm tt» informational benefits of evaluation efforts that 
inform <tecisionmakm about tb^ multiple f^ts of programs and how 
ttey function in (nagmatic ravinHiments. 

In this reiq:)^, the authors of chi^tm 2, 3, and 4 encourage the reaife.' 
to move toward evaluatic^is that take into accoiml the c(»nplexity of 
*^Iationships between program outcimies and program or;^uuzation. 
Studying high-jniority implonentaticm and outtrome issu^ within die 
mne historical period — a given plaining cycle — ^permits an evaluator to 
explore important interrelaticmships amfbg organizati<^ factoid, so- 
cial interventions, ami outeomes for a particular historical cohort of 
individuals exp(»^ to a pn^ram, and with ftie group of program «:tc^ 
who have (teveloped ami applied jm>gram policies within that same 
period. This aj^n^h provides tte importunity to acquire omsiderably 
nK»re useful infomudon tl^ can be olM^ned from isolated process and 
outcome stwlies condiKrted in difTerent periods unckr varying pn^ram 

A better uruterstanding of the intricate relati<nishi{s betw^ {m^gram 
implementation ar^i imi^ is directly responsive to the im^s of poli- 
cymakers« mtministrators, plaiu^is, and managers who must routinely 
ifteiiy {nioblems, (tevelc^ i^w polici^« ami modify programs. of 
their missicm is to determine which (Hogram changes are most af^rqm- 
ate in resolving problems and (kvelop stra^ies for making t}K>se 
diang^. In order to carry out this mi^icm sucx:essfully they must rely 
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on a broad, accmasc infonnadon base. 

The author of chapter 5 shifts our attention from the technical adjects 
of this exf^iKkd view of evaluation to die oi^ganizalional and political 
context and environment of slate and local evaluation activities. TTiis 
chapter <kfines significant pam^rship roles for the evaluator and 
pn^rmn planner in the evaluation process. It also explon;s some of tlK 
oi5ani2ati(Hial ami political l^rriers to evaluating, and sug^ts ways to 
reduce or woik around them. Most important, it dramatizes the impor- 
tance of assigning sufficient resources to the evaluaticm process, since 
the nature of those resources subtly direct and shape information 
production. 

Viewed as a whole, the interrelated chapters of the bock express the 
concept of evaluation introduced in chapter 1 , which defines evaluation 
as an umtertaking demanding a ccMiscious, purpc^ful use of scientiHc 
and organizational knowledge, skills, ami soisitivities. 
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Appendix 



This appendix is designed to provide midif ional information of interest 
ami use to tl^ reader. 

Appendix A affords an overview of the distinctive features of JTPA, 
tte cai^ example used throughout the book as an illustration to which 
general evaluation concepts are applied. 

Appendix B ^Jdrcsses an issue referred to in all cha^Mers but not fully 
develof— — the capacity of program MISs to support program evalu- 
adoiL 

Aji^iendix C outlii^ the series of evaluation guides produced by the 
JTPA Evaluation Design ftroject, on v hich the book is l^sed. 

Appendix D provides teckground information abr<ut the book's 
authors. 
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Appendix A 
The Case Example: 
Characteristics of Title H Iht^rams 
Under the Job IVaining Partnership Act 

Ann Bonar Blalock 



The application of general program evaluation concepts to a case 
example requires that the reader have an understanding of the 
characteristics of tl» illustrative program. In this section ihs key attrib- 
utes of JTPA are described, ctmsisient with the classif uion schen« 
used in discussing social programs in chapter 1. 
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Appendix B 
General Requirements of a JTPA 
MIS Supporting Evaluation 

David Grembowski 



Evaluation of social fMDgnuns requires management infonnation 
systrans (MISs) containing the necessary data for performing net impact, 
gross impact, and process evaluations. In tte first section of this suppte- 
ii«nt the general rcquirenwnts of an MIS suj^)mttng evaluation arc de- 
scribed. In the seomd section alternative MIS structures are discussed. 
Because the structure and craitent of an MIS tmy vary by jwogram, the 
Jc* Training Partror^p Act (JTPA) is used to illustrate these i^ues. 
The <fcsign of the system is perhaps more complex than for other social 
programs because it must satisfy the mfbrmation needs of three organi- 
zaUons: states, service delivery areas (SDAs), and subcontia;tore. 

General Requirements 

Six general requirements of a computerized JTPA MIS must be 
r>ati$fied if states, SDAs, and subcontractors are to perform impact and 
process evaluations of their respective programs: ( I ) satisfaction of data 
needs, (2) incorporation of an appropriate MIS structure, (3) attention to 
data communication, (4) data processing flexiWlity, (5) statistical soft- 
ware capability, and (6) skilled staff. Each requirement is discussed 
bek>w. 

Data Needs 

Impact models have specific information requirements. Net impact 
evaluation requires data from the JTPA MIS as well as other sources. The 
gross impact evaluation requires mainly JTPA MIS data, supplemented 
as needed by infonnation collected through participant and employer 
surveys Process evaluation uses a mixture of quantitative data from the 
MIS and both qualitative and quantitative data from other sources. The 
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JTPA MIS must contain, or have access to, data elements that satisfy 
these requiien^nts. 

Table B. 1 contains a list of the data elements, or variables, required to 
perfonn net and gross impact evaluations of JTPA. Unemployment 
insurance (UI) data limitatimis will likely jwevent most stales fixnn 
expanding the variable list for the state net impact evaluation. Local 
administrates, however, may wish to include other variables from the 
MIS in their gross impi»:t evaluations. 

Table B.l provic^ minimum data requirements. State and local 
ofTicials may add variables to the list as neected. hi either case, the 
computo-must contain sufficient storage capacity to record the variables 
over relevant periods for all participants included in the evaluation. In 
general, as the number of particij^nts and variables and their length of 
storage increase, so will the costs of maintaining the MIS. However, 
these costs can be offset by the benefits of new information, which these 
additional variables can produce in an evaluation. In constructing an MIS 
suitable for evaluatiim, state and local officials must seek a balance 
between the information needs of the evaluation and the various costs 
associated with satisfying those needs. 

The variables in table B. 1 must be ^nerated from data elancnts in the 
JTPA MIS. Each variable must be (tefmcd in the same manner acro^ all 
SDAs and subcoitractors in a stale. This is particulariy important for the 
net impact evaluation, where data from several SDAs are combined for 
analysis. If variables arc defuied differently across SDAs and subcon- 
tractors, the evaluation may produce erroneous conclusions. 

For example, wic variable in the MIS might be "classroom training." 
I . SDA I the classroom training variable contains a "1" for eveiy 
participant who receives this service. In SDA H, however, the service is 
defmed as classroom training plus job search assistance, aiid the class- 
room training variable contains a " 1" for every participant who receiver 
both services. The tkfinitions of classroom training in the two SDAs 
differ, which can lead to misleading results and conclusions in a iwt 
impact evaluation. For similar reasons, variables should also be ckfmed 
in the same manner whenever gross impact results of several SDAs are 
compared. 
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MIS Structure 

The structure, or configuration, of the MIS must su]^xm the evalu- 
atimi models. Two tosic MIS stitKnures exist: centralis] and decenmil- 
ized. Centrali%d structures usually coiKist of participant data for all 
SDAs gtored m a mainfiranie ccxminiter located at the ^e level (though 
some states have (tevdoped minicomputer systems). SDAs are usually 
cmmected to the mainfranw thnnigh terminals. peiMmal comfHrteis, or 
minicomputers. In some states SDAs have no acc^ to the state com- 
puter but receive reports on a periodic l»sis. Few subcontractors have 
access to state systems unless the subcofitrm:tor is a state agem^. 

In ctecentraUzed structwies, each SDA has cme a[ more persmal 
c(»n|Hiter or minicomputer (xmtalning its particif^t data. Tlx; ^e*s 
computer may or may not be linked to each SDA's computer. The 
system *s design and data definiti(»i5 may be establidied by the state, and 
both may become staiKlard across SDAs, Thus, the state and SDAs share 
control of the MIS; the state ccmtrols thnmgh system ctesign, vk^ile flie 
SDA controls thnmgh system operation. 

F^cij^t ami fmandd systems are usually separate in both central- 
ized and decoitralized ^iiHmuies. In fi^t, tl^ two systems scmietim^ 
exist on different mnputers. For example, some SDAs with a (tontml- 
ized participam system have fmancial data maintained by the state. 
While existing JTPA MIS structures are not bairiers to evaluation, their 
smictures must be taken into wrcour? ?n (Signing a prototype MIS to 
support the evaluation. 

CommunkatftHis 

Because implementation of JTPA is dispersed among state. SDA, and 
subcotitrwnor oiiganizBti<»is. information abmit **what goes €m'* in the 
program is also dispersed. In the JTPA MIS. agencies must have 
mechanisms for ccHnmunicating or transmitting data fmm oat ageiny to 
another. In SDAs that subcontriK:t intake, mechani«ns must exist for 
transmitting application and ^vollment data from tl» subccmtractor to 
the SDA, regardless of whether d» MIS has a centralis or (tecentral- 
VBcd structure. Different forms of data communicalion are possiMe: 
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TIk subccmtFurtOT enters tte ^i^icant data into its own com- 

IHiter aoA transmits it to the SDA by tetef^n^; 

The subcontractor is linked to the VTPA MIS and can enter 

applicant data directly into tt» MIS: or 

The subcontractor sends tt^ aj^ltcant forms to the SDA or state, 

whbh enters the data into ti^ MIS. 
Each subcontracts mu^ also be able to access its data in anc MIS. This 
is Msemial if subcontractors are to coiKluct gross impact and process 
evaluatifHis of tiKir own programs. Again, different MlS-to-ioibcontr^- 
trar cmnmunicaticHi modes are pmsible, sudi as a direct communicatiMi 
line with the MIS or monthly extracts wrinen m a floppy disk and mailed 
to the mibnmtntctor for analysis on its pa:iH>nal computer. 

Mechanisms must also exist fcM* data communicaticm between \hc 
SDA and die «tai£. In <^tralized MIS structures ei^ SDA must have die 
capability to enter aikl extnKt its data fn»n the state data base. In 
ctecentralized MIS structures the state must be able to extract data from 
tih£ SDA ccmiiHiter systems. Icteally, this is performed using telei^i^ 
lines or other c(»nmumcatiani channels link t)« SDA with tlw state 
MIS. Howrever, otho: forms of data commumcaticm are possible, such as 
monthly extractions of requested data on floppy disks that are mailed 
between the state and SDA. In short, in decentralized structures states 
need data from SDAs to perform state net impact evaluations; in 
cmtralized structures SDAs arai subcimtractorc need data fn»n the state 
to perfoim gross im|^t ami {Hoce^ evaluaticms of their own programs. 

TheK c»mmunicati(»i requirements apply to bath impact ami proc^ 
evaluatimis. Net imj^ evaluation, howev«-, i»s ukUiraial m]uirements; 
it also requires data frcan UI and wdfare infomiati(Hi systems. Assuming 
the net impact evaluation is perfcan^ at tl^ ^e level, ^ state c(mi|»iter 
syst^ must becfq^ble of accessing data fixnn tl;^ othn-syst»ns. If the 
JTPA, UI, and welfare data are all cm tlK san% antqniter, acc^is to tte 
^)f»t^ate data cm usually be re^ly fK:hieved. If ti^ data r^ide on 
different computers, the UI and welfare data must be transmitted to the 
JIPA MIS, using computn* tapes data ctmimunicaticHi chamois. 

Two issues usually (ktermine wl»tl»r inter^m:y data ctmmiunica- 
tion occurs. The Tost i^ue is cmtrd. That i*:, tl^ agency diitt controls tte 
data may be reluctant to release them to other agencies, reducing the 
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i^eiKies' ability to conduct evaluations of their own progmms. The 
sec(NKi issue is technical. In order for two comfniters to oHnmunicate, 
data must have standard formats, such as ASOI. Proper system itesign 
ami u% of the sanw l»and of computer equijm^t ageiunes can 
overcome this potential problem. 

Data Processing Flexibility 

All forms of evaluatifMi require the fn^ttom to manipulate and analyze 
data in a variety of ways. To satisfy this requirement tte JTPA MIS must 
employ software known as a data base management system (DBMS). In 
most computer systems in JTPA, data are distributed across several files. 
A DBMS can access da^ xrms files through relatively simple data 
retrieval cmnmarKls that can be applied in a wicte variety of data 
jHocessing environments. The commamls selectively pool information 
from the DBMS files into a form that satisfies the analyst's information 
iweds. Finther, a DBMS is adept at modifying files after they are created. 
Variables and records may be freely entered and deleted from previously 
developed files. In short, a DBMS provictes a flexible nKxle of data 
processing capable of addressing the infomnalion requirements of the 
evaluaticm. 

DBMS software commonly used on mainframes im;luctes ADABAS, 
DATACX)M, IDMA, IMS, SYSTEM 2000, TOTAL, ami several olhere. 
Personal computer DBMS software includes RB ASE 5000. REVELA- 
TION, DATAPLEX, DBASE, HEUX. ORACLE, and many othere. 
Each software {mckage has its own strengths ami w^kiwsses; ttey arc by 
no means equal. However, a JTPA MIS using DBMS software should 
provide the data processing flcxibiUty required by the evaluation. 

Some agcncfes may not have DBMS software in tlwir MIS, and the 
costs of adding tte software to their informatitm syioems may be 
jHohibitive. When a DBMS is not possible, a satisfactory alternative is 
to develop user-friendly, general purpose computer programs for ex- 
tracting data from ite data base. The u& t, who may be a computer 
programmer or a JTT*A administrator, supplies the jMograra with a list of 
(fesired data items and other parameters, and the i^Dgram retrieves the 
requested items from Uk data base and writes tJ«m onto an output file fw 
subM^uent analysis. 

C 

*J IS^ 
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Statistkai Software 

Although DBMS software is atkpi at maiupulaling data and generat- 
report lists, it does not have the capability of perfbnning the statistical 
analyses required by the impatn models. Therefore, the JTPA MIS 
should also include statu^ical software, such as SPSS, SAS, SYSTAT, 
CM" o&ser major tnwKi SPSS, for example, has developed a statistical 
packaj^ Uuu runs on most mainframes aiKi persona! computers. 

Skilled Stair 

Satisfying the above renJirements will be of little value if skilled staff 
is ittrt available to perform data i»t)ccssing. This docs not iKcessarily 
mean that mif members with computer scieiKc degn^s are neet^ for 
data processing to support evaluatiwj. In gross impact and process 
evaluaticHis, for example, the chief skill requirement is experience with 
DBMS and statistical software packages. States may wish to offer 
technical assistance to SDAs and subcontractors in the area of software 
use. The stale net impact evaluation, however, will probably require data 
processing personnel to combii^ the UI. welfare, and JTPA data sets into 
a fonn required for performing the evaluation. 

Types MIS Stmctiures 

In this guide "structure" refers to the compoiwnts of the infonnation 
system and how data are oiganized into files. The fomier may be one of 
two basic types, centralized or decentralized. 

Centralized MIS Structure 

A centralii^ surwnure is presented in exhibit 1 . Tt^ coitralized MIS 
features a mauiframe (or mini-) computer containing the JTPA MIS, 
located at the state level. The MIS uses DBMS aiKi statistical software. 
Tlie MIS cmtains tte panicii»nt system as well as data required for the 
benefit-cost analysis. The latter data are transmitted to tte st?te by each 
SDA, which operates its own financial system. However, in some states 
(such as tho&e with no SDAs) die financial system is either a part of the 
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centralized JTPA MIS or located on a se}>arate computer at tte state level. 
In ti» latter case a c^miunication interface links the JTPA MIS with tte 
financial system (if needed) as well as the UI aiKl welfare systems. As 
mentimed au1ier« this interface may be eitter a direct ctmimunication 
diannel or tape transfn*. 

Evaluation can occur at each level — state, SDA, and subcontractor. 
States can use the JTPA MIS to perform state process, gross, and net 
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isnpasl evaluations. SDAs and subcantradors can perfonn jHiXfiss and 
gross imp^l cvaluatkms of their respective pn^rams. In this case, 
communicalion links ccmnect the si«c JTPA MIS with aU SDAs ami in 
some cases selected contractors, such as a local Job Service office. 
DifTeient links may exist, such as follows: 

Local ofiiccs use terminals or PCs to wxess the data base, and all 
analyses arc performed on the mainframe computer. CcHnmwnicalion 
ocairs ♦hrou^i telephone lines (or electronic medium). Security 
controls in the DBH" pennit :9ch SDA to access only its data. The 
DBMS does nxA allov SDAs cither lo delete data from the data base or 
to modify cxisUng records. Thus, while SDAs and subcontractors can 
add new records to the data base, they can only "read" data after they are 
entered. 

Telephone \inc^ (or other electronic medium) are used to transfer data 
firm the state MIS to the SD/..* or subomtraclor's PC or minicomputer. 

Each month state provides each SDA with a flqppy disk(s) 
cont^ning all data entered into the MIS during the period. SDAs analyze 
the data on their own PCs or minicomputers. 

Periodic reports, one method of statc-to-local data transfer, are not 
included btxause they do not satisfy the information requirements of the 
local evaluation. 

Different types of SDA-subcontractw communication channels exist 
as shown in exhibit B.l. SDA I provides its subcontractors only with 
pp.per reports; s ubcontractors can wnly perform cmdc evaluarions of their 
IMOgrams. After receiving its data from the state, SDA II relays appropri- 
ate data to each subcontractor using flo^Jy disks. Subcontr^tors per- 
form their own e .valuations using tlieir own PCs. In SDA UI the job 
service subcwitiactw h-^ a direct ctMnmunication line to the SDA's 
computer for accessing its data. In short, if subconlractoni are to gain 
access to MIS data in most states, the data must first be transferre-i to the 

local level (usua'Iy the SDA), and the SDA must then grant its subcon- 
tractors access to the data through one mechanism or another. Thus, 
ahbough data redundancy is inevitable under this arrangement, it gives 
service jwoviders the information they need to evaluate their programs. 
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DecentraHied MIS Structure 

The disUoguishing features of the decmralized MIS structure are that 
(1) each SDA operates its own MIS. and (2) communication channels 
link SDA computer systems with the state (sec exhibit B^) ). SDA data 
are transmitted to the state either over telej^one lines cm- through mail 
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deliveiy of floppy disks. The siaic computer has inltrfeces wiA the UI 
iiml welfare dair. bases for perfcwming net impact evaJuatiwis. 

A decoitralized MIS can be cn^ded in ^veral ways, as stw?wn in 
exhiWl B.2. In SDA I a minicomputer holds its JTPA MIS, which 
incli»k!s ti» DBMS for the particii^t and financial systms as well as 
statisticd software. The minicomputer has "multi-user softwaie*" that 
allows subcontr^ors and the state to lu^ss the data base simultane- 
ously through terminals or PCs. These agencies cwmnunicate with the 
minicomiNiter using a telei^cnw and a modem. 

SDA n also operates a minicomputer, but it does not permit outside 
»xess to ti» data base. However, the state aiKJ subcontractors regularly 
request data from die MIS, whidi the SDA provides on floppy disks. 

The bottom half of exhilnt B.2 presents an SDA MIS using persomU 
ccHTiputers ami a local area iKtwork. Altlraugh participant and fmancial 
systons are separate, both data sets are stored on a single hard disk. (The 
fiiumcial system could be located on a different cranputer.) The size of 
the disk varies with the size of the SDA, but disks with 50 to 80 
megabytes of storage should be adequate for most SDAs. 

Piefsonal computers located at the SDA, subcontr^tor, and state levels 
form a "local area network"; each PC in the networic gains access to the 
data base through the nctwoik*s "file server.** The file server, which is 
actually a PC with local area iwtwork software, acts as the gateknper. It 
regulates access to the data base throughout the mtwmk. Using a 
telephone modem, state officials and subcontractors with PCs can enter 
the network and access the data base. Each PC must use common DBMS 
aixi financial software to gain entry. 

File Structure 

Di^^ereni file structures are possible in the JTPA MIS data base. Only 
<m£ file structure is (kscribed in this section; it can be us^ in bolh 
^traliz^ aiKi decentralized s^tems. JTPA data l^ses in most smtes 
have more ccwnplcx file structures than the one described here. Our intent 
is not to describe the ideal JTPA MIS, but rather to identify elements that 
are essential to performing evaluation. For evaluation purposes the JT1>A 
MIS contains the following six files: 
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1. Paitidpant Master File (containing application and terminaticm 
infomuuicm). 

2. Participant Service Hie (containing training ai^ su{^)on service 
informatifNi). 

3. Participant FoIlow-Up File (containing information <m e^h foUow- 
up). 

4. Employer Master File (containing information on local employers). 

5. Staff Master File (craitaimng infomiation on SDA and subccmtractor 
staff who serve paiticipants). 

6. Subccmtr»;tor Master Hie (ccmtainingiunformationtm SDA suboMV- 
tractors). 

The DBMS uses common identifiers to interrelate data in one file with 
data in another file. For exfunple, if the Puticipant Master File and the 
Participant Service File both contain the participant's ID, the DBMS can 
interrelate master file data with service file data. This is essential to 
p^oiming gross impact evaluation, where we are interested in correlat- 
ing the services particifMuits receive (Service Rle) with their outcon^ 
(Participant Master File). 

Program administrators s(»netlme$ wi^ to supplement program data 
witih additional information collected through sp^ial surveys of clients 
or other sources. These data can usually be »kled to th^ data l^ise as a 
separate file i^ysically, but linked with other files through common 
identifiers. 



Appendix C 
Evaluation Guides Produced by 
the JTPA Evaluation Design Project 

Tbs ccmiprehaisive concept of evaluatkm demonstrated in tlw JTPA 
Evaluation Design Project and expressed in the Project's series of 
evaluati(Mi guides and regional evaluation workshqw required an inier- 
disdplinaiy torni ccmimitled to ahigh level of intellectual interaction and 
the production of research tools that were both competent and useful . The 
team that was brought together to prepare the series of volumes below 
combined expertise from economics, sociology, planning and social 
work. 



Utieof Vohiim 


AuUnkt 


I. 


OVERVIEW 


Ann Bonar Byock and 
DebcM^ I^ldman 


II. 


A GENERAL PLANNING GUIDE 


Deborah Feldman 


III. 


A GUIDE FOR NET IMPACT EVALUATIONS 


Tcny R. Johnson 


IV. 


AN IMPLEMENTATION MANUAL TOR NET 
IMPACT EVALUATIONS 


Terry R. Johnson 


V. 


A GUIDE FOR GROSS IMPACT EVALUATIONS 


Car! Simpson 


Vl. 


A GUIDE FOR PROCESS EVALUATIONS 


David Grembowski 


vn. 


ISSUES IN EVALUATING COSTS AND 
BENEFITS 


Ernst W. Stn»nsdorfer 


VII. 


MIS ISSUES IN EVALUATING J FPA 


David Grembowski 


vni. THE DEBATE OVER QUASI-EXPERIMENTAL 


Ann BtMiar Bklock 



VS. EXPERIMENTAL DESIGNS 



Note: These volumes are actcssible through the ERIC CkaringhcHLse microfiche sctjcs 
in public libraries. 
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Aim Bonar Blalock Is Special Assistant to tlK^ I>eputy Commissioner^ 
Washington State Employment S^urity Def^rtment. Her assignments 
have involved polkry analysis, evaluation research, and tte design of 
cfemonstration projects in employment ami training, work/welfare, and 
human services integrttion. She was respcH^ible for developing the 
ct»K^ for The JTPA Evaluation Design Project, acquiring fimds to 
support it, and initially directing tl^ Pto^t. She subisequently served as 
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of volumes. His evaluation iMckground is the most extensive of 
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In dbapfer S, Debcnah Fekiman shifts the reader's attemira to d% 
ctevdk^»natt of a ci^^ralnlity to evaluate ptograms, and to (Hganizadonal 
ai^ pdltical issi»s NimHmding tl» initiatiim, plaimmg, iinplementatiim 
and use of evaliiations at the state and local level. Woiidng on sp&M 
phuining, techmcal training and evaluaticm projects for the D^uty , 
CMnmissiooer's Office, Washington State Employmail Security De- 
partment, she a:pmized ami directed the Fioject*s regional evaluation 
tminii^ w(»1csl»ps, and contributed extensively to Evaluatim Forum. 
Ste has broad experioKe in oiganizatimial devdc^^moit, planning and 
project ttevelopment in the areas of worker dislocation, economic devel- 
opment and wcmioi's issues, and has pii^lisli^ in the area of plant 
closures and worker readjustment. Her graduate academic training is in 
planning. 
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